Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 14:11:55.268257 23497 test_util.cc:268] Using random seed: 1297176632
I20240503 14:11:55.279191 23497 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:11:55.279273 23497 ts_itest-base.cc:113] --------------
I20240503 14:11:55.279315 23497 ts_itest-base.cc:114] 6 tablet servers
I20240503 14:11:55.279357 23497 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:11:55.279403 23497 ts_itest-base.cc:116] --------------
2024-05-03T14:11:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:11:55Z Disabled control of system clock
I20240503 14:11:55.304425 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.242.126:33953
--webserver_interface=127.22.242.126
--webserver_port=0
--builtin_ntp_servers=127.22.242.84:37849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.242.126:33953
--location_mapping_cmd=/tmp/dist-test-taskqfBHnl/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 14:11:55.621417 23509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:55.621730 23509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:55.621816 23509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:55.629329 23509 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:55.629455 23509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:11:55.629547 23509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:55.629622 23509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:11:55.629683 23509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:11:55.638687 23509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:37849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskqfBHnl/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.22.242.126:33953
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.242.126:33953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.242.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:55.639958 23509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:55.641683 23509 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:55.651978 23514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:55.651985 23517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:55.651985 23515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:55.652462 23509 server_base.cc:1034] running on GCE node
I20240503 14:11:55.653363 23509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:55.654706 23509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:55.655922 23509 hybrid_clock.cc:648] HybridClock initialized: now 1714745515655867 us; error 44 us; skew 500 ppm
I20240503 14:11:55.656301 23509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:55.662631 23509 webserver.cc:460] Webserver started at http://127.22.242.126:43775/ using document root <none> and password file <none>
I20240503 14:11:55.663342 23509 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:55.663456 23509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:55.663746 23509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:55.666296 23509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "45604633057f43e9ad77fa7396eacc2b"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-r3pd"
I20240503 14:11:55.666998 23509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "45604633057f43e9ad77fa7396eacc2b"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-r3pd"
I20240503 14:11:55.671907 23509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240503 14:11:55.675282 23523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:55.676595 23509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:11:55.676756 23509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
uuid: "45604633057f43e9ad77fa7396eacc2b"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-r3pd"
I20240503 14:11:55.676972 23509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:55.692471 23509 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:55.693296 23509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:55.693552 23509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:55.711118 23509 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.126:33953
I20240503 14:11:55.711164 23574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.126:33953 every 8 connection(s)
I20240503 14:11:55.712507 23509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:11:55.716205 23575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:55.721267 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 23509
I20240503 14:11:55.721504 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:11:55.725642 23575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b: Bootstrap starting.
I20240503 14:11:55.728951 23575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:55.729929 23575 log.cc:824] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b: Log is configured to *not* fsync() on all Append() calls
I20240503 14:11:55.732285 23575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b: No bootstrap required, opened a new log
I20240503 14:11:55.739305 23575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45604633057f43e9ad77fa7396eacc2b" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 33953 } }
I20240503 14:11:55.739813 23575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:55.739944 23575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45604633057f43e9ad77fa7396eacc2b, State: Initialized, Role: FOLLOWER
I20240503 14:11:55.740588 23575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45604633057f43e9ad77fa7396eacc2b" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 33953 } }
I20240503 14:11:55.740773 23575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:11:55.740900 23575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:11:55.741039 23575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:55.744032 23575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45604633057f43e9ad77fa7396eacc2b" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 33953 } }
I20240503 14:11:55.744491 23575 leader_election.cc:302] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45604633057f43e9ad77fa7396eacc2b; no voters:
I20240503 14:11:55.745184 23575 leader_election.cc:288] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:11:55.745328 23580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:55.746007 23580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b [term 1 LEADER]: Becoming Leader. State: Replica: 45604633057f43e9ad77fa7396eacc2b, State: Running, Role: LEADER
I20240503 14:11:55.746625 23580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45604633057f43e9ad77fa7396eacc2b" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 33953 } }
I20240503 14:11:55.747859 23575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:11:55.750578 23582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45604633057f43e9ad77fa7396eacc2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45604633057f43e9ad77fa7396eacc2b" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 33953 } } }
I20240503 14:11:55.750401 23581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45604633057f43e9ad77fa7396eacc2b. Latest consensus state: current_term: 1 leader_uuid: "45604633057f43e9ad77fa7396eacc2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45604633057f43e9ad77fa7396eacc2b" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 33953 } } }
I20240503 14:11:55.751564 23582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b [sys.catalog]: This master's current role is: LEADER
I20240503 14:11:55.751569 23581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b [sys.catalog]: This master's current role is: LEADER
I20240503 14:11:55.752364 23585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:11:55.767685 23585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:11:55.776921 23585 catalog_manager.cc:1347] Generated new cluster ID: 61e3d66bdb4a403297b89828c702b2d9
I20240503 14:11:55.777050 23585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:11:55.790963 23585 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:11:55.792675 23585 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:11:55.810729 23585 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 45604633057f43e9ad77fa7396eacc2b: Generated new TSK 0
I20240503 14:11:55.811856 23585 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:11:55.827458 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.65:0
--local_ip_for_outbound_sockets=127.22.242.65
--webserver_interface=127.22.242.65
--webserver_port=0
--tserver_master_addrs=127.22.242.126:33953
--builtin_ntp_servers=127.22.242.84:37849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:11:56.174188 23599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:56.174474 23599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:56.174588 23599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:56.182662 23599 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:56.182793 23599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:56.182945 23599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.65
I20240503 14:11:56.192546 23599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:37849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.242.65
--webserver_port=0
--tserver_master_addrs=127.22.242.126:33953
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.65
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:56.193931 23599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:56.195636 23599 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:56.207562 23604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.208072 23607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.208248 23605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:56.209208 23599 server_base.cc:1034] running on GCE node
I20240503 14:11:56.209775 23599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:56.210685 23599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:56.211880 23599 hybrid_clock.cc:648] HybridClock initialized: now 1714745516211833 us; error 44 us; skew 500 ppm
I20240503 14:11:56.212235 23599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:56.219877 23599 webserver.cc:460] Webserver started at http://127.22.242.65:39319/ using document root <none> and password file <none>
I20240503 14:11:56.220718 23599 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:56.220841 23599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:56.221158 23599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:56.223800 23599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "8795a3c2a7014baf9126dea0a6ce1bb4"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-r3pd"
I20240503 14:11:56.224642 23599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "8795a3c2a7014baf9126dea0a6ce1bb4"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-r3pd"
I20240503 14:11:56.229930 23599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 14:11:56.233489 23613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.234833 23599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240503 14:11:56.234992 23599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "8795a3c2a7014baf9126dea0a6ce1bb4"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-r3pd"
I20240503 14:11:56.235193 23599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:56.254942 23599 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:56.255810 23599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:56.256085 23599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:56.257091 23599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:56.258651 23599 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:56.258749 23599 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.258857 23599 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:56.258929 23599 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.292811 23599 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.65:44175
I20240503 14:11:56.292873 23725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.65:44175 every 8 connection(s)
I20240503 14:11:56.294119 23599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:11:56.295264 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 23599
I20240503 14:11:56.295487 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:11:56.299353 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.66:0
--local_ip_for_outbound_sockets=127.22.242.66
--webserver_interface=127.22.242.66
--webserver_port=0
--tserver_master_addrs=127.22.242.126:33953
--builtin_ntp_servers=127.22.242.84:37849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:11:56.309856 23726 heartbeater.cc:347] Connected to a master server at 127.22.242.126:33953
I20240503 14:11:56.310201 23726 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:56.311154 23726 heartbeater.cc:511] Master 127.22.242.126:33953 requested a full tablet report, sending...
I20240503 14:11:56.376468 23540 ts_manager.cc:194] Registered new tserver with Master: 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175)
I20240503 14:11:56.379032 23540 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.65:46847
W20240503 14:11:56.661132 23730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:56.661459 23730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:56.661545 23730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:56.668972 23730 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:56.669091 23730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:56.669215 23730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.66
I20240503 14:11:56.678493 23730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:37849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.242.66
--webserver_port=0
--tserver_master_addrs=127.22.242.126:33953
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.66
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:56.679817 23730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:56.681630 23730 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:56.693991 23737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.694469 23736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.695196 23739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:56.696455 23730 server_base.cc:1034] running on GCE node
I20240503 14:11:56.697090 23730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:56.698133 23730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:56.699343 23730 hybrid_clock.cc:648] HybridClock initialized: now 1714745516699293 us; error 47 us; skew 500 ppm
I20240503 14:11:56.699647 23730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:56.701857 23730 webserver.cc:460] Webserver started at http://127.22.242.66:39729/ using document root <none> and password file <none>
I20240503 14:11:56.702603 23730 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:56.702718 23730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:56.703011 23730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:56.705612 23730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d4eea540f3c241c1ba6290ca53769163"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-r3pd"
I20240503 14:11:56.706350 23730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d4eea540f3c241c1ba6290ca53769163"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-r3pd"
I20240503 14:11:56.711570 23730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20240503 14:11:56.715118 23745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.716631 23730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 14:11:56.716795 23730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d4eea540f3c241c1ba6290ca53769163"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-r3pd"
I20240503 14:11:56.717012 23730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:56.730806 23730 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:56.731602 23730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:56.731863 23730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:56.732918 23730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:56.734386 23730 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:56.734472 23730 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.734577 23730 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:56.734642 23730 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.767802 23730 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.66:46473
I20240503 14:11:56.767848 23857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.66:46473 every 8 connection(s)
I20240503 14:11:56.769244 23730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:11:56.779441 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 23730
I20240503 14:11:56.779625 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:11:56.782419 23858 heartbeater.cc:347] Connected to a master server at 127.22.242.126:33953
I20240503 14:11:56.782729 23858 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:56.782917 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.67:0
--local_ip_for_outbound_sockets=127.22.242.67
--webserver_interface=127.22.242.67
--webserver_port=0
--tserver_master_addrs=127.22.242.126:33953
--builtin_ntp_servers=127.22.242.84:37849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:11:56.783588 23858 heartbeater.cc:511] Master 127.22.242.126:33953 requested a full tablet report, sending...
I20240503 14:11:56.828305 23539 ts_manager.cc:194] Registered new tserver with Master: d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473)
I20240503 14:11:56.829264 23539 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.66:44783
W20240503 14:11:57.126379 23861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:57.126698 23861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:57.126842 23861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:57.134303 23861 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:57.134469 23861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:57.134636 23861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.67
I20240503 14:11:57.144220 23861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:37849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.242.67
--webserver_port=0
--tserver_master_addrs=127.22.242.126:33953
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.67
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:57.145579 23861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:57.147310 23861 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:57.158963 23867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.159221 23870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.159260 23868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:57.160435 23861 server_base.cc:1034] running on GCE node
I20240503 14:11:57.161027 23861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:57.161942 23861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:57.163130 23861 hybrid_clock.cc:648] HybridClock initialized: now 1714745517163081 us; error 40 us; skew 500 ppm
I20240503 14:11:57.163437 23861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:57.165652 23861 webserver.cc:460] Webserver started at http://127.22.242.67:35293/ using document root <none> and password file <none>
I20240503 14:11:57.166397 23861 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:57.166512 23861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:57.166828 23861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:57.169426 23861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d40aeb06cfcc4c3a897b40290d12a15b"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-r3pd"
I20240503 14:11:57.170187 23861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d40aeb06cfcc4c3a897b40290d12a15b"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-r3pd"
I20240503 14:11:57.175930 23861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240503 14:11:57.179200 23876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.180621 23861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:11:57.180792 23861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d40aeb06cfcc4c3a897b40290d12a15b"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-r3pd"
I20240503 14:11:57.180982 23861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:57.195667 23861 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:57.196578 23861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:57.196866 23861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:57.197840 23861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:57.199303 23861 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:57.199398 23861 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.199504 23861 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:57.199573 23861 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.233000 23861 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.67:37697
I20240503 14:11:57.233058 23988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.67:37697 every 8 connection(s)
I20240503 14:11:57.234275 23861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:11:57.240324 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 23861
I20240503 14:11:57.240531 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:11:57.244189 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.68:0
--local_ip_for_outbound_sockets=127.22.242.68
--webserver_interface=127.22.242.68
--webserver_port=0
--tserver_master_addrs=127.22.242.126:33953
--builtin_ntp_servers=127.22.242.84:37849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:11:57.248664 23989 heartbeater.cc:347] Connected to a master server at 127.22.242.126:33953
I20240503 14:11:57.249032 23989 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:57.250016 23989 heartbeater.cc:511] Master 127.22.242.126:33953 requested a full tablet report, sending...
I20240503 14:11:57.295652 23539 ts_manager.cc:194] Registered new tserver with Master: d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697)
I20240503 14:11:57.296972 23539 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.67:34677
I20240503 14:11:57.382582 23726 heartbeater.cc:503] Master 127.22.242.126:33953 was elected leader, sending a full tablet report...
W20240503 14:11:57.586107 23992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:57.586441 23992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:57.586582 23992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:57.594324 23992 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:57.594457 23992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:57.594630 23992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.68
I20240503 14:11:57.604502 23992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:37849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.242.68
--webserver_port=0
--tserver_master_addrs=127.22.242.126:33953
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.68
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:57.605841 23992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:57.607573 23992 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:57.619593 23998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.619575 23999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.619911 24001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:57.620347 23992 server_base.cc:1034] running on GCE node
I20240503 14:11:57.620931 23992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:57.621870 23992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:57.623111 23992 hybrid_clock.cc:648] HybridClock initialized: now 1714745517623009 us; error 51 us; skew 500 ppm
I20240503 14:11:57.623412 23992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:57.625620 23992 webserver.cc:460] Webserver started at http://127.22.242.68:33207/ using document root <none> and password file <none>
I20240503 14:11:57.626363 23992 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:57.626474 23992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:57.626762 23992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:57.629393 23992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1e79cc5c647f476c98101f9e48b22b9e"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-r3pd"
I20240503 14:11:57.630128 23992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1e79cc5c647f476c98101f9e48b22b9e"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-r3pd"
I20240503 14:11:57.635114 23992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240503 14:11:57.638411 24007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.639714 23992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240503 14:11:57.639861 23992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1e79cc5c647f476c98101f9e48b22b9e"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-r3pd"
I20240503 14:11:57.640046 23992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:57.671813 23992 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:57.672693 23992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:57.672966 23992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:57.673902 23992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:57.675331 23992 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:57.675424 23992 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.675529 23992 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:57.675598 23992 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.710314 23992 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.68:37675
I20240503 14:11:57.710433 24119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.68:37675 every 8 connection(s)
I20240503 14:11:57.711999 23992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:11:57.713688 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 23992
I20240503 14:11:57.713856 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:11:57.717284 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.69:0
--local_ip_for_outbound_sockets=127.22.242.69
--webserver_interface=127.22.242.69
--webserver_port=0
--tserver_master_addrs=127.22.242.126:33953
--builtin_ntp_servers=127.22.242.84:37849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:11:57.728029 24120 heartbeater.cc:347] Connected to a master server at 127.22.242.126:33953
I20240503 14:11:57.728456 24120 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:57.729606 24120 heartbeater.cc:511] Master 127.22.242.126:33953 requested a full tablet report, sending...
I20240503 14:11:57.788795 23539 ts_manager.cc:194] Registered new tserver with Master: 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675)
I20240503 14:11:57.789739 23539 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.68:55551
I20240503 14:11:57.832500 23858 heartbeater.cc:503] Master 127.22.242.126:33953 was elected leader, sending a full tablet report...
W20240503 14:11:58.076511 24123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:58.076798 24123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:58.076892 24123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:58.084686 24123 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:58.084837 24123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:58.084959 24123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.69
I20240503 14:11:58.094131 24123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:37849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.242.69
--webserver_port=0
--tserver_master_addrs=127.22.242.126:33953
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.69
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:58.095399 24123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:58.097131 24123 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:58.109092 24130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.109234 24129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.109234 24132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:58.109911 24123 server_base.cc:1034] running on GCE node
I20240503 14:11:58.110489 24123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:58.111387 24123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:58.112578 24123 hybrid_clock.cc:648] HybridClock initialized: now 1714745518112517 us; error 55 us; skew 500 ppm
I20240503 14:11:58.112886 24123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:58.115123 24123 webserver.cc:460] Webserver started at http://127.22.242.69:34055/ using document root <none> and password file <none>
I20240503 14:11:58.115840 24123 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:58.115953 24123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:58.116303 24123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:58.119405 24123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c9a573f6faf64131b114af1e33c1b308"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-r3pd"
I20240503 14:11:58.120187 24123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c9a573f6faf64131b114af1e33c1b308"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-r3pd"
I20240503 14:11:58.125211 24123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240503 14:11:58.128499 24138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.129793 24123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:11:58.129943 24123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c9a573f6faf64131b114af1e33c1b308"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-r3pd"
I20240503 14:11:58.130136 24123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:58.148550 24123 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:58.149363 24123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:58.149622 24123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:58.150532 24123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:58.151919 24123 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:58.152015 24123 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.152139 24123 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:58.152235 24123 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20240503 14:11:58.187315 24123 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.69:45475
I20240503 14:11:58.187366 24250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.69:45475 every 8 connection(s)
I20240503 14:11:58.188714 24123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:11:58.196518 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 24123
I20240503 14:11:58.197309 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:11:58.201129 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.70:0
--local_ip_for_outbound_sockets=127.22.242.70
--webserver_interface=127.22.242.70
--webserver_port=0
--tserver_master_addrs=127.22.242.126:33953
--builtin_ntp_servers=127.22.242.84:37849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:11:58.202052 24251 heartbeater.cc:347] Connected to a master server at 127.22.242.126:33953
I20240503 14:11:58.202344 24251 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:58.203122 24251 heartbeater.cc:511] Master 127.22.242.126:33953 requested a full tablet report, sending...
I20240503 14:11:58.263226 23539 ts_manager.cc:194] Registered new tserver with Master: c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475)
I20240503 14:11:58.264225 23539 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.69:60955
I20240503 14:11:58.299793 23989 heartbeater.cc:503] Master 127.22.242.126:33953 was elected leader, sending a full tablet report...
W20240503 14:11:58.555603 24254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:58.555924 24254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:58.556013 24254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:58.563670 24254 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:58.563808 24254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:58.563923 24254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.70
I20240503 14:11:58.573520 24254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:37849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.242.70
--webserver_port=0
--tserver_master_addrs=127.22.242.126:33953
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.70
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:58.574836 24254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:58.576722 24254 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:58.589788 24263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.589955 24260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.589915 24261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:58.590821 24254 server_base.cc:1034] running on GCE node
I20240503 14:11:58.591439 24254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:58.592434 24254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:58.593627 24254 hybrid_clock.cc:648] HybridClock initialized: now 1714745518593577 us; error 41 us; skew 500 ppm
I20240503 14:11:58.593946 24254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:58.596282 24254 webserver.cc:460] Webserver started at http://127.22.242.70:42801/ using document root <none> and password file <none>
I20240503 14:11:58.597014 24254 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:58.597118 24254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:58.597447 24254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:58.600005 24254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "ff53511345114c4ab97a9a8decef3a74"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-r3pd"
I20240503 14:11:58.600782 24254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "ff53511345114c4ab97a9a8decef3a74"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-r3pd"
I20240503 14:11:58.605741 24254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 14:11:58.609185 24269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.610522 24254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:11:58.610682 24254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "ff53511345114c4ab97a9a8decef3a74"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-r3pd"
I20240503 14:11:58.610889 24254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:58.647827 24254 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:58.649019 24254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:58.649360 24254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:58.650607 24254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:58.652590 24254 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:58.652696 24254 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.652810 24254 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:58.652877 24254 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.688558 24254 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.70:35067
I20240503 14:11:58.688645 24381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.70:35067 every 8 connection(s)
I20240503 14:11:58.689818 24254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:11:58.699278 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 24254
I20240503 14:11:58.699450 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 14:11:58.703449 24382 heartbeater.cc:347] Connected to a master server at 127.22.242.126:33953
I20240503 14:11:58.703766 24382 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:58.704574 24382 heartbeater.cc:511] Master 127.22.242.126:33953 requested a full tablet report, sending...
I20240503 14:11:58.782244 23539 ts_manager.cc:194] Registered new tserver with Master: ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067)
I20240503 14:11:58.783174 23539 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.70:42265
I20240503 14:11:58.789968 23497 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 14:11:58.792752 24120 heartbeater.cc:503] Master 127.22.242.126:33953 was elected leader, sending a full tablet report...
I20240503 14:11:58.810047 23497 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 14:11:59.267323 24251 heartbeater.cc:503] Master 127.22.242.126:33953 was elected leader, sending a full tablet report...
I20240503 14:11:59.786224 24382 heartbeater.cc:503] Master 127.22.242.126:33953 was elected leader, sending a full tablet report...
I20240503 14:12:02.564129 23539 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33990:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:02.623580 24055 tablet_service.cc:1467] Processing CreateTablet for tablet 3b562d2ac0524913b7eaded7a0441958 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c148a0e5d13e4c10a014d28ebd23da64]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.624786 23923 tablet_service.cc:1467] Processing CreateTablet for tablet cf6dcac0c61c4c5fa12add8298c29dcf (DEFAULT_TABLE table=rebalance_test_table_0 [id=c148a0e5d13e4c10a014d28ebd23da64]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.626214 23923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6dcac0c61c4c5fa12add8298c29dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.626489 24055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b562d2ac0524913b7eaded7a0441958. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.623450 23924 tablet_service.cc:1467] Processing CreateTablet for tablet e0a3e437577742b1807b24a193ebc82b (DEFAULT_TABLE table=rebalance_test_table_0 [id=c148a0e5d13e4c10a014d28ebd23da64]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.632462 23924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a3e437577742b1807b24a193ebc82b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.633571 24317 tablet_service.cc:1467] Processing CreateTablet for tablet e0a3e437577742b1807b24a193ebc82b (DEFAULT_TABLE table=rebalance_test_table_0 [id=c148a0e5d13e4c10a014d28ebd23da64]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.635113 24317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a3e437577742b1807b24a193ebc82b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.641012 23793 tablet_service.cc:1467] Processing CreateTablet for tablet e0a3e437577742b1807b24a193ebc82b (DEFAULT_TABLE table=rebalance_test_table_0 [id=c148a0e5d13e4c10a014d28ebd23da64]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.642706 23793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a3e437577742b1807b24a193ebc82b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.649156 23661 tablet_service.cc:1467] Processing CreateTablet for tablet 3b562d2ac0524913b7eaded7a0441958 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c148a0e5d13e4c10a014d28ebd23da64]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.650192 23660 tablet_service.cc:1467] Processing CreateTablet for tablet cf6dcac0c61c4c5fa12add8298c29dcf (DEFAULT_TABLE table=rebalance_test_table_0 [id=c148a0e5d13e4c10a014d28ebd23da64]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.651214 23661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b562d2ac0524913b7eaded7a0441958. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.653539 23660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6dcac0c61c4c5fa12add8298c29dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.653388 24186 tablet_service.cc:1467] Processing CreateTablet for tablet 3b562d2ac0524913b7eaded7a0441958 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c148a0e5d13e4c10a014d28ebd23da64]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.654888 24186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b562d2ac0524913b7eaded7a0441958. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.658481 24185 tablet_service.cc:1467] Processing CreateTablet for tablet cf6dcac0c61c4c5fa12add8298c29dcf (DEFAULT_TABLE table=rebalance_test_table_0 [id=c148a0e5d13e4c10a014d28ebd23da64]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.659134 24185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6dcac0c61c4c5fa12add8298c29dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.677445 24409 tablet_bootstrap.cc:492] T cf6dcac0c61c4c5fa12add8298c29dcf P c9a573f6faf64131b114af1e33c1b308: Bootstrap starting.
I20240503 14:12:02.679471 24411 tablet_bootstrap.cc:492] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4: Bootstrap starting.
I20240503 14:12:02.684039 24411 tablet_bootstrap.cc:654] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.685279 24406 tablet_bootstrap.cc:492] T 3b562d2ac0524913b7eaded7a0441958 P 1e79cc5c647f476c98101f9e48b22b9e: Bootstrap starting.
I20240503 14:12:02.687352 24408 tablet_bootstrap.cc:492] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74: Bootstrap starting.
I20240503 14:12:02.691030 24408 tablet_bootstrap.cc:654] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.692867 24410 tablet_bootstrap.cc:492] T e0a3e437577742b1807b24a193ebc82b P d4eea540f3c241c1ba6290ca53769163: Bootstrap starting.
I20240503 14:12:02.697149 24411 log.cc:824] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:02.697373 24409 tablet_bootstrap.cc:654] T cf6dcac0c61c4c5fa12add8298c29dcf P c9a573f6faf64131b114af1e33c1b308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.697607 24406 tablet_bootstrap.cc:654] T 3b562d2ac0524913b7eaded7a0441958 P 1e79cc5c647f476c98101f9e48b22b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.698781 24409 log.cc:824] T cf6dcac0c61c4c5fa12add8298c29dcf P c9a573f6faf64131b114af1e33c1b308: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:02.698838 24406 log.cc:824] T 3b562d2ac0524913b7eaded7a0441958 P 1e79cc5c647f476c98101f9e48b22b9e: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:02.700181 24407 tablet_bootstrap.cc:492] T e0a3e437577742b1807b24a193ebc82b P d40aeb06cfcc4c3a897b40290d12a15b: Bootstrap starting.
I20240503 14:12:02.702575 24411 tablet_bootstrap.cc:492] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4: No bootstrap required, opened a new log
I20240503 14:12:02.702931 24411 ts_tablet_manager.cc:1367] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.000s
I20240503 14:12:02.703207 24410 tablet_bootstrap.cc:654] T e0a3e437577742b1807b24a193ebc82b P d4eea540f3c241c1ba6290ca53769163: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.704030 24407 tablet_bootstrap.cc:654] T e0a3e437577742b1807b24a193ebc82b P d40aeb06cfcc4c3a897b40290d12a15b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.704648 24410 log.cc:824] T e0a3e437577742b1807b24a193ebc82b P d4eea540f3c241c1ba6290ca53769163: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:02.705847 24407 log.cc:824] T e0a3e437577742b1807b24a193ebc82b P d40aeb06cfcc4c3a897b40290d12a15b: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:02.706367 24408 log.cc:824] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:02.709298 24410 tablet_bootstrap.cc:492] T e0a3e437577742b1807b24a193ebc82b P d4eea540f3c241c1ba6290ca53769163: No bootstrap required, opened a new log
I20240503 14:12:02.709300 24409 tablet_bootstrap.cc:492] T cf6dcac0c61c4c5fa12add8298c29dcf P c9a573f6faf64131b114af1e33c1b308: No bootstrap required, opened a new log
I20240503 14:12:02.709682 24409 ts_tablet_manager.cc:1367] T cf6dcac0c61c4c5fa12add8298c29dcf P c9a573f6faf64131b114af1e33c1b308: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.000s
I20240503 14:12:02.717242 24406 tablet_bootstrap.cc:492] T 3b562d2ac0524913b7eaded7a0441958 P 1e79cc5c647f476c98101f9e48b22b9e: No bootstrap required, opened a new log
I20240503 14:12:02.717629 24406 ts_tablet_manager.cc:1367] T 3b562d2ac0524913b7eaded7a0441958 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.010s
I20240503 14:12:02.725190 24410 ts_tablet_manager.cc:1367] T e0a3e437577742b1807b24a193ebc82b P d4eea540f3c241c1ba6290ca53769163: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.000s
I20240503 14:12:02.731572 24411 raft_consensus.cc:348] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:02.733101 24411 raft_consensus.cc:374] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.733274 24411 raft_consensus.cc:729] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.733394 24408 tablet_bootstrap.cc:492] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74: No bootstrap required, opened a new log
I20240503 14:12:02.733712 24408 ts_tablet_manager.cc:1367] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74: Time spent bootstrapping tablet: real 0.047s user 0.015s sys 0.012s
I20240503 14:12:02.734344 24411 consensus_queue.cc:260] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:02.736459 24406 raft_consensus.cc:348] T 3b562d2ac0524913b7eaded7a0441958 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:02.737851 24407 tablet_bootstrap.cc:492] T e0a3e437577742b1807b24a193ebc82b P d40aeb06cfcc4c3a897b40290d12a15b: No bootstrap required, opened a new log
I20240503 14:12:02.738293 24407 ts_tablet_manager.cc:1367] T e0a3e437577742b1807b24a193ebc82b P d40aeb06cfcc4c3a897b40290d12a15b: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.000s
I20240503 14:12:02.739535 24409 raft_consensus.cc:348] T cf6dcac0c61c4c5fa12add8298c29dcf P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:02.740243 24409 raft_consensus.cc:374] T cf6dcac0c61c4c5fa12add8298c29dcf P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.740478 24409 raft_consensus.cc:729] T cf6dcac0c61c4c5fa12add8298c29dcf P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.741381 24409 consensus_queue.cc:260] T cf6dcac0c61c4c5fa12add8298c29dcf P c9a573f6faf64131b114af1e33c1b308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:02.742267 24406 raft_consensus.cc:374] T 3b562d2ac0524913b7eaded7a0441958 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.742617 24406 raft_consensus.cc:729] T 3b562d2ac0524913b7eaded7a0441958 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.742671 24411 ts_tablet_manager.cc:1398] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent starting tablet: real 0.040s user 0.018s sys 0.004s
I20240503 14:12:02.744817 24411 tablet_bootstrap.cc:492] T cf6dcac0c61c4c5fa12add8298c29dcf P 8795a3c2a7014baf9126dea0a6ce1bb4: Bootstrap starting.
I20240503 14:12:02.746575 24411 tablet_bootstrap.cc:654] T cf6dcac0c61c4c5fa12add8298c29dcf P 8795a3c2a7014baf9126dea0a6ce1bb4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.743986 24406 consensus_queue.cc:260] T 3b562d2ac0524913b7eaded7a0441958 P 1e79cc5c647f476c98101f9e48b22b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:02.749934 24410 raft_consensus.cc:348] T e0a3e437577742b1807b24a193ebc82b P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:02.750382 24410 raft_consensus.cc:374] T e0a3e437577742b1807b24a193ebc82b P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.750480 24410 raft_consensus.cc:729] T e0a3e437577742b1807b24a193ebc82b P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.751143 24410 consensus_queue.cc:260] T e0a3e437577742b1807b24a193ebc82b P d4eea540f3c241c1ba6290ca53769163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:02.752878 24411 tablet_bootstrap.cc:492] T cf6dcac0c61c4c5fa12add8298c29dcf P 8795a3c2a7014baf9126dea0a6ce1bb4: No bootstrap required, opened a new log
I20240503 14:12:02.752657 24408 raft_consensus.cc:348] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:02.753108 24411 ts_tablet_manager.cc:1367] T cf6dcac0c61c4c5fa12add8298c29dcf P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20240503 14:12:02.753093 24408 raft_consensus.cc:374] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.753201 24408 raft_consensus.cc:729] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.753839 24408 consensus_queue.cc:260] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:02.754139 24409 ts_tablet_manager.cc:1398] T cf6dcac0c61c4c5fa12add8298c29dcf P c9a573f6faf64131b114af1e33c1b308: Time spent starting tablet: real 0.044s user 0.004s sys 0.009s
I20240503 14:12:02.747767 24407 raft_consensus.cc:348] T e0a3e437577742b1807b24a193ebc82b P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:02.754283 24407 raft_consensus.cc:374] T e0a3e437577742b1807b24a193ebc82b P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.754451 24407 raft_consensus.cc:729] T e0a3e437577742b1807b24a193ebc82b P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.754412 24411 raft_consensus.cc:348] T cf6dcac0c61c4c5fa12add8298c29dcf P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:02.754729 24411 raft_consensus.cc:374] T cf6dcac0c61c4c5fa12add8298c29dcf P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.754876 24411 raft_consensus.cc:729] T cf6dcac0c61c4c5fa12add8298c29dcf P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.755179 24411 consensus_queue.cc:260] T cf6dcac0c61c4c5fa12add8298c29dcf P 8795a3c2a7014baf9126dea0a6ce1bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:02.755234 24407 consensus_queue.cc:260] T e0a3e437577742b1807b24a193ebc82b P d40aeb06cfcc4c3a897b40290d12a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:02.760036 24409 tablet_bootstrap.cc:492] T 3b562d2ac0524913b7eaded7a0441958 P c9a573f6faf64131b114af1e33c1b308: Bootstrap starting.
I20240503 14:12:02.761884 24411 ts_tablet_manager.cc:1398] T cf6dcac0c61c4c5fa12add8298c29dcf P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:12:02.762305 24409 tablet_bootstrap.cc:654] T 3b562d2ac0524913b7eaded7a0441958 P c9a573f6faf64131b114af1e33c1b308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.764237 24407 ts_tablet_manager.cc:1398] T e0a3e437577742b1807b24a193ebc82b P d40aeb06cfcc4c3a897b40290d12a15b: Time spent starting tablet: real 0.026s user 0.011s sys 0.007s
I20240503 14:12:02.764714 24407 tablet_bootstrap.cc:492] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b: Bootstrap starting.
I20240503 14:12:02.766701 24407 tablet_bootstrap.cc:654] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.774875 24406 ts_tablet_manager.cc:1398] T 3b562d2ac0524913b7eaded7a0441958 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent starting tablet: real 0.057s user 0.011s sys 0.005s
I20240503 14:12:02.775038 24407 tablet_bootstrap.cc:492] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b: No bootstrap required, opened a new log
I20240503 14:12:02.775298 24407 ts_tablet_manager.cc:1367] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20240503 14:12:02.776466 24407 raft_consensus.cc:348] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:02.776759 24407 raft_consensus.cc:374] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.776921 24407 raft_consensus.cc:729] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.777251 24407 consensus_queue.cc:260] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:02.779845 24407 ts_tablet_manager.cc:1398] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:02.780279 24409 tablet_bootstrap.cc:492] T 3b562d2ac0524913b7eaded7a0441958 P c9a573f6faf64131b114af1e33c1b308: No bootstrap required, opened a new log
I20240503 14:12:02.780596 24409 ts_tablet_manager.cc:1367] T 3b562d2ac0524913b7eaded7a0441958 P c9a573f6faf64131b114af1e33c1b308: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.005s
I20240503 14:12:02.782094 24409 raft_consensus.cc:348] T 3b562d2ac0524913b7eaded7a0441958 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:02.782979 24409 raft_consensus.cc:374] T 3b562d2ac0524913b7eaded7a0441958 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.783152 24409 raft_consensus.cc:729] T 3b562d2ac0524913b7eaded7a0441958 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.783522 24425 raft_consensus.cc:482] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.783514 24409 consensus_queue.cc:260] T 3b562d2ac0524913b7eaded7a0441958 P c9a573f6faf64131b114af1e33c1b308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:02.783828 24425 raft_consensus.cc:504] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:02.785413 24410 ts_tablet_manager.cc:1398] T e0a3e437577742b1807b24a193ebc82b P d4eea540f3c241c1ba6290ca53769163: Time spent starting tablet: real 0.060s user 0.020s sys 0.006s
I20240503 14:12:02.786130 24409 ts_tablet_manager.cc:1398] T 3b562d2ac0524913b7eaded7a0441958 P c9a573f6faf64131b114af1e33c1b308: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:12:02.786813 24408 ts_tablet_manager.cc:1398] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74: Time spent starting tablet: real 0.053s user 0.009s sys 0.018s
I20240503 14:12:02.788240 24425 leader_election.cc:288] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473), d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697)
I20240503 14:12:02.798029 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a3e437577742b1807b24a193ebc82b" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" is_pre_election: true
I20240503 14:12:02.798031 23813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a3e437577742b1807b24a193ebc82b" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163" is_pre_election: true
I20240503 14:12:02.798489 23944 raft_consensus.cc:2454] T e0a3e437577742b1807b24a193ebc82b P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 0.
I20240503 14:12:02.798485 23813 raft_consensus.cc:2454] T e0a3e437577742b1807b24a193ebc82b P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 0.
I20240503 14:12:02.799115 24273 leader_election.cc:302] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:02.799510 24425 raft_consensus.cc:2781] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.799647 24425 raft_consensus.cc:482] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.799749 24425 raft_consensus.cc:3037] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:02.800055 23727 tablet.cc:2369] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:02.801817 24425 raft_consensus.cc:504] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:02.802417 24425 leader_election.cc:288] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 election: Requested vote from peers d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473), d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697)
I20240503 14:12:02.802899 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a3e437577742b1807b24a193ebc82b" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b"
I20240503 14:12:02.802907 23813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a3e437577742b1807b24a193ebc82b" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163"
I20240503 14:12:02.803072 23944 raft_consensus.cc:3037] T e0a3e437577742b1807b24a193ebc82b P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.803098 23813 raft_consensus.cc:3037] T e0a3e437577742b1807b24a193ebc82b P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.805243 23813 raft_consensus.cc:2454] T e0a3e437577742b1807b24a193ebc82b P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 1.
I20240503 14:12:02.805590 23944 raft_consensus.cc:2454] T e0a3e437577742b1807b24a193ebc82b P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 1.
I20240503 14:12:02.805657 24273 leader_election.cc:302] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:02.805944 24425 raft_consensus.cc:2781] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.806105 24425 raft_consensus.cc:686] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [term 1 LEADER]: Becoming Leader. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Running, Role: LEADER
I20240503 14:12:02.806581 24425 consensus_queue.cc:237] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:02.811468 23538 catalog_manager.cc:5484] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff53511345114c4ab97a9a8decef3a74 (127.22.242.70). New cstate: current_term: 1 leader_uuid: "ff53511345114c4ab97a9a8decef3a74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:02.855543 24424 raft_consensus.cc:482] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.855751 24424 raft_consensus.cc:504] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:02.857400 24424 leader_election.cc:288] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475), 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175)
I20240503 14:12:02.866847 23681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6dcac0c61c4c5fa12add8298c29dcf" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" is_pre_election: true
I20240503 14:12:02.866843 24206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6dcac0c61c4c5fa12add8298c29dcf" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308" is_pre_election: true
I20240503 14:12:02.867409 24206 raft_consensus.cc:2454] T cf6dcac0c61c4c5fa12add8298c29dcf P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 0.
I20240503 14:12:02.867409 23681 raft_consensus.cc:2454] T cf6dcac0c61c4c5fa12add8298c29dcf P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 0.
I20240503 14:12:02.867991 23878 leader_election.cc:302] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:02.868397 24424 raft_consensus.cc:2781] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.868518 24424 raft_consensus.cc:482] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.868609 24424 raft_consensus.cc:3037] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.871109 24424 raft_consensus.cc:504] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:02.871274 24418 raft_consensus.cc:482] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.871435 24418 raft_consensus.cc:504] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:02.871840 24424 leader_election.cc:288] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 election: Requested vote from peers c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475), 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175)
I20240503 14:12:02.872305 23681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6dcac0c61c4c5fa12add8298c29dcf" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4"
I20240503 14:12:02.872287 24206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6dcac0c61c4c5fa12add8298c29dcf" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308"
I20240503 14:12:02.872488 23681 raft_consensus.cc:3037] T cf6dcac0c61c4c5fa12add8298c29dcf P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.872483 24206 raft_consensus.cc:3037] T cf6dcac0c61c4c5fa12add8298c29dcf P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.873507 24418 leader_election.cc:288] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675), c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475)
I20240503 14:12:02.875798 24206 raft_consensus.cc:2454] T cf6dcac0c61c4c5fa12add8298c29dcf P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 1.
I20240503 14:12:02.876430 23681 raft_consensus.cc:2454] T cf6dcac0c61c4c5fa12add8298c29dcf P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 1.
I20240503 14:12:02.876667 23878 leader_election.cc:302] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:02.877103 24424 raft_consensus.cc:2781] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.878069 24424 raft_consensus.cc:686] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [term 1 LEADER]: Becoming Leader. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Running, Role: LEADER
I20240503 14:12:02.879066 24424 consensus_queue.cc:237] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:02.884891 23538 catalog_manager.cc:5484] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b reported cstate change: term changed from 0 to 1, leader changed from <none> to d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67). New cstate: current_term: 1 leader_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:02.885187 24075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b562d2ac0524913b7eaded7a0441958" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e" is_pre_election: true
I20240503 14:12:02.885668 24075 raft_consensus.cc:2454] T 3b562d2ac0524913b7eaded7a0441958 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 0.
I20240503 14:12:02.886313 23617 leader_election.cc:302] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, 8795a3c2a7014baf9126dea0a6ce1bb4; no voters:
I20240503 14:12:02.886780 24418 raft_consensus.cc:2781] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.886912 24418 raft_consensus.cc:482] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.887015 24418 raft_consensus.cc:3037] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.886991 24206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b562d2ac0524913b7eaded7a0441958" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308" is_pre_election: true
I20240503 14:12:02.887238 24206 raft_consensus.cc:2454] T 3b562d2ac0524913b7eaded7a0441958 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 0.
I20240503 14:12:02.889429 24418 raft_consensus.cc:504] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:02.890113 24418 leader_election.cc:288] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 election: Requested vote from peers 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675), c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475)
I20240503 14:12:02.890666 24075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b562d2ac0524913b7eaded7a0441958" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e"
I20240503 14:12:02.890875 24075 raft_consensus.cc:3037] T 3b562d2ac0524913b7eaded7a0441958 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.890987 24206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b562d2ac0524913b7eaded7a0441958" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308"
I20240503 14:12:02.891218 24206 raft_consensus.cc:3037] T 3b562d2ac0524913b7eaded7a0441958 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.893496 24075 raft_consensus.cc:2454] T 3b562d2ac0524913b7eaded7a0441958 P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 1.
I20240503 14:12:02.893551 24206 raft_consensus.cc:2454] T 3b562d2ac0524913b7eaded7a0441958 P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 1.
I20240503 14:12:02.893883 23617 leader_election.cc:302] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, 8795a3c2a7014baf9126dea0a6ce1bb4; no voters:
I20240503 14:12:02.894182 24418 raft_consensus.cc:2781] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.894573 24418 raft_consensus.cc:686] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 LEADER]: Becoming Leader. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Running, Role: LEADER
I20240503 14:12:02.895161 24418 consensus_queue.cc:237] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:02.900032 23538 catalog_manager.cc:5484] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:02.943593 24252 tablet.cc:2369] T 3b562d2ac0524913b7eaded7a0441958 P c9a573f6faf64131b114af1e33c1b308: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:12:02.944522 24383 tablet.cc:2369] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:12:02.968025 24121 tablet.cc:2369] T 3b562d2ac0524913b7eaded7a0441958 P 1e79cc5c647f476c98101f9e48b22b9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:12:02.993108 23990 tablet.cc:2369] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:12:03.025053 23859 tablet.cc:2369] T e0a3e437577742b1807b24a193ebc82b P d4eea540f3c241c1ba6290ca53769163: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:03.273906 24424 consensus_queue.cc:1035] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.283898 24424 consensus_queue.cc:1035] T cf6dcac0c61c4c5fa12add8298c29dcf P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.290949 24425 consensus_queue.cc:1035] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.298964 24418 consensus_queue.cc:1035] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.329823 24418 consensus_queue.cc:1035] T 3b562d2ac0524913b7eaded7a0441958 P 8795a3c2a7014baf9126dea0a6ce1bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.343931 24425 consensus_queue.cc:1035] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:03.359411 24439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:03.359736 24439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:03.368012 24439 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 317.432 ms
time per row: 0.158716 ms
I20240503 14:12:03.832329 23536 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33990:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:03.840714 24055 tablet_service.cc:1467] Processing CreateTablet for tablet ac35b944198044e1af026f24d35e0b7b (DEFAULT_TABLE table=rebalance_test_table_1 [id=2d8bf83f56064dc7b4afbdf2c5adbc46]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.840991 24317 tablet_service.cc:1467] Processing CreateTablet for tablet ac35b944198044e1af026f24d35e0b7b (DEFAULT_TABLE table=rebalance_test_table_1 [id=2d8bf83f56064dc7b4afbdf2c5adbc46]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.841182 24055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac35b944198044e1af026f24d35e0b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.841398 24317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac35b944198044e1af026f24d35e0b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.841780 24316 tablet_service.cc:1467] Processing CreateTablet for tablet d31c3fbce3a74c32aff549e9d1ceda13 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2d8bf83f56064dc7b4afbdf2c5adbc46]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.842307 24316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d31c3fbce3a74c32aff549e9d1ceda13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.842731 23924 tablet_service.cc:1467] Processing CreateTablet for tablet f1c9262f9e154fe5b30c2ec66b26e4d5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2d8bf83f56064dc7b4afbdf2c5adbc46]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.843142 23924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1c9262f9e154fe5b30c2ec66b26e4d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.848554 23792 tablet_service.cc:1467] Processing CreateTablet for tablet d31c3fbce3a74c32aff549e9d1ceda13 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2d8bf83f56064dc7b4afbdf2c5adbc46]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.848664 23793 tablet_service.cc:1467] Processing CreateTablet for tablet ac35b944198044e1af026f24d35e0b7b (DEFAULT_TABLE table=rebalance_test_table_1 [id=2d8bf83f56064dc7b4afbdf2c5adbc46]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.849123 23792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d31c3fbce3a74c32aff549e9d1ceda13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.849442 23660 tablet_service.cc:1467] Processing CreateTablet for tablet f1c9262f9e154fe5b30c2ec66b26e4d5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2d8bf83f56064dc7b4afbdf2c5adbc46]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.849844 23660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1c9262f9e154fe5b30c2ec66b26e4d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.849951 24054 tablet_service.cc:1467] Processing CreateTablet for tablet d31c3fbce3a74c32aff549e9d1ceda13 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2d8bf83f56064dc7b4afbdf2c5adbc46]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.850504 24054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d31c3fbce3a74c32aff549e9d1ceda13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.852845 24185 tablet_service.cc:1467] Processing CreateTablet for tablet f1c9262f9e154fe5b30c2ec66b26e4d5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2d8bf83f56064dc7b4afbdf2c5adbc46]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.853232 24185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1c9262f9e154fe5b30c2ec66b26e4d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.853335 24516 tablet_bootstrap.cc:492] T ac35b944198044e1af026f24d35e0b7b P 1e79cc5c647f476c98101f9e48b22b9e: Bootstrap starting.
I20240503 14:12:03.855263 24516 tablet_bootstrap.cc:654] T ac35b944198044e1af026f24d35e0b7b P 1e79cc5c647f476c98101f9e48b22b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.856369 23793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac35b944198044e1af026f24d35e0b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.868615 24521 tablet_bootstrap.cc:492] T ac35b944198044e1af026f24d35e0b7b P ff53511345114c4ab97a9a8decef3a74: Bootstrap starting.
I20240503 14:12:03.868902 24520 tablet_bootstrap.cc:492] T d31c3fbce3a74c32aff549e9d1ceda13 P d4eea540f3c241c1ba6290ca53769163: Bootstrap starting.
I20240503 14:12:03.870852 24522 tablet_bootstrap.cc:492] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308: Bootstrap starting.
I20240503 14:12:03.870855 24521 tablet_bootstrap.cc:654] T ac35b944198044e1af026f24d35e0b7b P ff53511345114c4ab97a9a8decef3a74: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.872020 24520 tablet_bootstrap.cc:654] T d31c3fbce3a74c32aff549e9d1ceda13 P d4eea540f3c241c1ba6290ca53769163: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.872713 24519 tablet_bootstrap.cc:492] T f1c9262f9e154fe5b30c2ec66b26e4d5 P d40aeb06cfcc4c3a897b40290d12a15b: Bootstrap starting.
I20240503 14:12:03.873206 24517 tablet_bootstrap.cc:492] T f1c9262f9e154fe5b30c2ec66b26e4d5 P 8795a3c2a7014baf9126dea0a6ce1bb4: Bootstrap starting.
I20240503 14:12:03.874137 24522 tablet_bootstrap.cc:654] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.875355 24517 tablet_bootstrap.cc:654] T f1c9262f9e154fe5b30c2ec66b26e4d5 P 8795a3c2a7014baf9126dea0a6ce1bb4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.875928 24520 tablet_bootstrap.cc:492] T d31c3fbce3a74c32aff549e9d1ceda13 P d4eea540f3c241c1ba6290ca53769163: No bootstrap required, opened a new log
I20240503 14:12:03.876083 24520 ts_tablet_manager.cc:1367] T d31c3fbce3a74c32aff549e9d1ceda13 P d4eea540f3c241c1ba6290ca53769163: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:12:03.876713 24516 tablet_bootstrap.cc:492] T ac35b944198044e1af026f24d35e0b7b P 1e79cc5c647f476c98101f9e48b22b9e: No bootstrap required, opened a new log
I20240503 14:12:03.876869 24516 ts_tablet_manager.cc:1367] T ac35b944198044e1af026f24d35e0b7b P 1e79cc5c647f476c98101f9e48b22b9e: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.009s
I20240503 14:12:03.877197 24519 tablet_bootstrap.cc:654] T f1c9262f9e154fe5b30c2ec66b26e4d5 P d40aeb06cfcc4c3a897b40290d12a15b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.877120 24520 raft_consensus.cc:348] T d31c3fbce3a74c32aff549e9d1ceda13 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:03.877374 24520 raft_consensus.cc:374] T d31c3fbce3a74c32aff549e9d1ceda13 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.877487 24520 raft_consensus.cc:729] T d31c3fbce3a74c32aff549e9d1ceda13 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.877801 24520 consensus_queue.cc:260] T d31c3fbce3a74c32aff549e9d1ceda13 P d4eea540f3c241c1ba6290ca53769163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:03.877889 24516 raft_consensus.cc:348] T ac35b944198044e1af026f24d35e0b7b P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:03.878114 24516 raft_consensus.cc:374] T ac35b944198044e1af026f24d35e0b7b P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.878221 24516 raft_consensus.cc:729] T ac35b944198044e1af026f24d35e0b7b P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.878242 24521 tablet_bootstrap.cc:492] T ac35b944198044e1af026f24d35e0b7b P ff53511345114c4ab97a9a8decef3a74: No bootstrap required, opened a new log
I20240503 14:12:03.878504 24521 ts_tablet_manager.cc:1367] T ac35b944198044e1af026f24d35e0b7b P ff53511345114c4ab97a9a8decef3a74: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 14:12:03.878454 24516 consensus_queue.cc:260] T ac35b944198044e1af026f24d35e0b7b P 1e79cc5c647f476c98101f9e48b22b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:03.879825 24521 raft_consensus.cc:348] T ac35b944198044e1af026f24d35e0b7b P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:03.880187 24521 raft_consensus.cc:374] T ac35b944198044e1af026f24d35e0b7b P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.880349 24521 raft_consensus.cc:729] T ac35b944198044e1af026f24d35e0b7b P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.880685 24521 consensus_queue.cc:260] T ac35b944198044e1af026f24d35e0b7b P ff53511345114c4ab97a9a8decef3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:03.881362 24521 ts_tablet_manager.cc:1398] T ac35b944198044e1af026f24d35e0b7b P ff53511345114c4ab97a9a8decef3a74: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:03.881776 24521 tablet_bootstrap.cc:492] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74: Bootstrap starting.
I20240503 14:12:03.884102 24522 tablet_bootstrap.cc:492] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308: No bootstrap required, opened a new log
I20240503 14:12:03.884301 24522 ts_tablet_manager.cc:1367] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.004s
I20240503 14:12:03.884851 24520 ts_tablet_manager.cc:1398] T d31c3fbce3a74c32aff549e9d1ceda13 P d4eea540f3c241c1ba6290ca53769163: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240503 14:12:03.885329 24520 tablet_bootstrap.cc:492] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163: Bootstrap starting.
I20240503 14:12:03.885191 24522 raft_consensus.cc:348] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:03.885426 24522 raft_consensus.cc:374] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.885530 24522 raft_consensus.cc:729] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.885955 24521 tablet_bootstrap.cc:654] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.885797 24522 consensus_queue.cc:260] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:03.888598 24517 tablet_bootstrap.cc:492] T f1c9262f9e154fe5b30c2ec66b26e4d5 P 8795a3c2a7014baf9126dea0a6ce1bb4: No bootstrap required, opened a new log
I20240503 14:12:03.888886 24517 ts_tablet_manager.cc:1367] T f1c9262f9e154fe5b30c2ec66b26e4d5 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240503 14:12:03.889818 24517 raft_consensus.cc:348] T f1c9262f9e154fe5b30c2ec66b26e4d5 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:03.890211 24517 raft_consensus.cc:374] T f1c9262f9e154fe5b30c2ec66b26e4d5 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.890347 24520 tablet_bootstrap.cc:654] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.890373 24517 raft_consensus.cc:729] T f1c9262f9e154fe5b30c2ec66b26e4d5 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.890753 24517 consensus_queue.cc:260] T f1c9262f9e154fe5b30c2ec66b26e4d5 P 8795a3c2a7014baf9126dea0a6ce1bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:03.891073 24522 ts_tablet_manager.cc:1398] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20240503 14:12:03.892848 24517 ts_tablet_manager.cc:1398] T f1c9262f9e154fe5b30c2ec66b26e4d5 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:03.892884 24519 tablet_bootstrap.cc:492] T f1c9262f9e154fe5b30c2ec66b26e4d5 P d40aeb06cfcc4c3a897b40290d12a15b: No bootstrap required, opened a new log
I20240503 14:12:03.893038 24519 ts_tablet_manager.cc:1367] T f1c9262f9e154fe5b30c2ec66b26e4d5 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.007s
I20240503 14:12:03.894062 24519 raft_consensus.cc:348] T f1c9262f9e154fe5b30c2ec66b26e4d5 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:03.894342 24519 raft_consensus.cc:374] T f1c9262f9e154fe5b30c2ec66b26e4d5 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.894445 24519 raft_consensus.cc:729] T f1c9262f9e154fe5b30c2ec66b26e4d5 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.896306 24519 consensus_queue.cc:260] T f1c9262f9e154fe5b30c2ec66b26e4d5 P d40aeb06cfcc4c3a897b40290d12a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:03.897121 24519 ts_tablet_manager.cc:1398] T f1c9262f9e154fe5b30c2ec66b26e4d5 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:03.897271 24521 tablet_bootstrap.cc:492] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74: No bootstrap required, opened a new log
I20240503 14:12:03.897504 24521 ts_tablet_manager.cc:1367] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240503 14:12:03.897877 24520 tablet_bootstrap.cc:492] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163: No bootstrap required, opened a new log
I20240503 14:12:03.898059 24520 ts_tablet_manager.cc:1367] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20240503 14:12:03.898576 24521 raft_consensus.cc:348] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:03.898947 24521 raft_consensus.cc:374] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.898859 24520 raft_consensus.cc:348] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:03.899096 24520 raft_consensus.cc:374] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.899128 24521 raft_consensus.cc:729] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.899494 24521 consensus_queue.cc:260] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:03.899780 24520 raft_consensus.cc:729] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.900754 24521 ts_tablet_manager.cc:1398] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:03.901087 24516 ts_tablet_manager.cc:1398] T ac35b944198044e1af026f24d35e0b7b P 1e79cc5c647f476c98101f9e48b22b9e: Time spent starting tablet: real 0.024s user 0.002s sys 0.011s
I20240503 14:12:03.901629 24516 tablet_bootstrap.cc:492] T d31c3fbce3a74c32aff549e9d1ceda13 P 1e79cc5c647f476c98101f9e48b22b9e: Bootstrap starting.
I20240503 14:12:03.902333 24520 consensus_queue.cc:260] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:03.903014 24520 ts_tablet_manager.cc:1398] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 14:12:03.904271 24516 tablet_bootstrap.cc:654] T d31c3fbce3a74c32aff549e9d1ceda13 P 1e79cc5c647f476c98101f9e48b22b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.906347 24516 tablet_bootstrap.cc:492] T d31c3fbce3a74c32aff549e9d1ceda13 P 1e79cc5c647f476c98101f9e48b22b9e: No bootstrap required, opened a new log
I20240503 14:12:03.906508 24516 ts_tablet_manager.cc:1367] T d31c3fbce3a74c32aff549e9d1ceda13 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240503 14:12:03.907316 24516 raft_consensus.cc:348] T d31c3fbce3a74c32aff549e9d1ceda13 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:03.907512 24516 raft_consensus.cc:374] T d31c3fbce3a74c32aff549e9d1ceda13 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.907593 24516 raft_consensus.cc:729] T d31c3fbce3a74c32aff549e9d1ceda13 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.907829 24516 consensus_queue.cc:260] T d31c3fbce3a74c32aff549e9d1ceda13 P 1e79cc5c647f476c98101f9e48b22b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:03.908427 24516 ts_tablet_manager.cc:1398] T d31c3fbce3a74c32aff549e9d1ceda13 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 14:12:03.909921 24530 raft_consensus.cc:482] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:03.910106 24530 raft_consensus.cc:504] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:03.912225 24530 leader_election.cc:288] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697), 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175)
I20240503 14:12:03.919793 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1c9262f9e154fe5b30c2ec66b26e4d5" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" is_pre_election: true
I20240503 14:12:03.919962 23681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1c9262f9e154fe5b30c2ec66b26e4d5" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" is_pre_election: true
I20240503 14:12:03.920080 23944 raft_consensus.cc:2454] T f1c9262f9e154fe5b30c2ec66b26e4d5 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 0.
I20240503 14:12:03.920257 23681 raft_consensus.cc:2454] T f1c9262f9e154fe5b30c2ec66b26e4d5 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 0.
I20240503 14:12:03.920701 24139 leader_election.cc:302] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:03.921180 24530 raft_consensus.cc:2781] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:03.921303 24530 raft_consensus.cc:482] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:03.921409 24530 raft_consensus.cc:3037] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.923748 24530 raft_consensus.cc:504] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:03.924374 24530 leader_election.cc:288] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 election: Requested vote from peers d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697), 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175)
I20240503 14:12:03.924811 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1c9262f9e154fe5b30c2ec66b26e4d5" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b"
I20240503 14:12:03.925014 23944 raft_consensus.cc:3037] T f1c9262f9e154fe5b30c2ec66b26e4d5 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.924969 23681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1c9262f9e154fe5b30c2ec66b26e4d5" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4"
I20240503 14:12:03.925176 23681 raft_consensus.cc:3037] T f1c9262f9e154fe5b30c2ec66b26e4d5 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.926961 23681 raft_consensus.cc:2454] T f1c9262f9e154fe5b30c2ec66b26e4d5 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 1.
I20240503 14:12:03.926971 23944 raft_consensus.cc:2454] T f1c9262f9e154fe5b30c2ec66b26e4d5 P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 1.
I20240503 14:12:03.927394 24140 leader_election.cc:302] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8795a3c2a7014baf9126dea0a6ce1bb4, c9a573f6faf64131b114af1e33c1b308; no voters:
I20240503 14:12:03.927713 24530 raft_consensus.cc:2781] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:03.928196 24530 raft_consensus.cc:686] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [term 1 LEADER]: Becoming Leader. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Running, Role: LEADER
I20240503 14:12:03.928730 24530 consensus_queue.cc:237] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:03.932441 23536 catalog_manager.cc:5484] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9a573f6faf64131b114af1e33c1b308 (127.22.242.69). New cstate: current_term: 1 leader_uuid: "c9a573f6faf64131b114af1e33c1b308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:03.946377 24425 raft_consensus.cc:482] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:03.946583 24425 raft_consensus.cc:504] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:03.947292 24425 leader_election.cc:288] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473), 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675)
I20240503 14:12:03.947868 23812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d31c3fbce3a74c32aff549e9d1ceda13" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163" is_pre_election: true
I20240503 14:12:03.948222 23812 raft_consensus.cc:2454] T d31c3fbce3a74c32aff549e9d1ceda13 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 0.
I20240503 14:12:03.948750 24273 leader_election.cc:302] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:03.949015 24425 raft_consensus.cc:2781] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:03.949165 24425 raft_consensus.cc:482] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:03.949302 24425 raft_consensus.cc:3037] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.952018 24425 raft_consensus.cc:504] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:03.952626 24425 leader_election.cc:288] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 election: Requested vote from peers d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473), 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675)
I20240503 14:12:03.953073 23812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d31c3fbce3a74c32aff549e9d1ceda13" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163"
I20240503 14:12:03.953250 23812 raft_consensus.cc:3037] T d31c3fbce3a74c32aff549e9d1ceda13 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.955292 24075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d31c3fbce3a74c32aff549e9d1ceda13" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e" is_pre_election: true
I20240503 14:12:03.955543 24075 raft_consensus.cc:2454] T d31c3fbce3a74c32aff549e9d1ceda13 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 0.
I20240503 14:12:03.955425 24074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d31c3fbce3a74c32aff549e9d1ceda13" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e"
I20240503 14:12:03.955771 24074 raft_consensus.cc:3037] T d31c3fbce3a74c32aff549e9d1ceda13 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.955881 23812 raft_consensus.cc:2454] T d31c3fbce3a74c32aff549e9d1ceda13 P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 1.
I20240503 14:12:03.956580 24273 leader_election.cc:302] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:03.957077 24425 raft_consensus.cc:2781] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:03.957260 24425 raft_consensus.cc:686] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [term 1 LEADER]: Becoming Leader. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Running, Role: LEADER
I20240503 14:12:03.957521 24425 consensus_queue.cc:237] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:03.958336 24074 raft_consensus.cc:2454] T d31c3fbce3a74c32aff549e9d1ceda13 P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 1.
I20240503 14:12:03.960480 23536 catalog_manager.cc:5484] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff53511345114c4ab97a9a8decef3a74 (127.22.242.70). New cstate: current_term: 1 leader_uuid: "ff53511345114c4ab97a9a8decef3a74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:04.275949 24528 raft_consensus.cc:482] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:04.276145 24528 raft_consensus.cc:504] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:04.277634 24528 leader_election.cc:288] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675), ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067)
I20240503 14:12:04.285027 24529 raft_consensus.cc:482] T ac35b944198044e1af026f24d35e0b7b P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:04.285285 24074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac35b944198044e1af026f24d35e0b7b" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e" is_pre_election: true
I20240503 14:12:04.285295 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac35b944198044e1af026f24d35e0b7b" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74" is_pre_election: true
I20240503 14:12:04.285272 24529 raft_consensus.cc:504] T ac35b944198044e1af026f24d35e0b7b P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:04.285634 24337 raft_consensus.cc:2454] T ac35b944198044e1af026f24d35e0b7b P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 0.
I20240503 14:12:04.285889 24074 raft_consensus.cc:2454] T ac35b944198044e1af026f24d35e0b7b P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 0.
I20240503 14:12:04.286238 23747 leader_election.cc:302] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:04.286654 24528 raft_consensus.cc:2781] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:04.286800 24528 raft_consensus.cc:482] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:04.286895 24528 raft_consensus.cc:3037] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.287101 24529 leader_election.cc:288] T ac35b944198044e1af026f24d35e0b7b P 1e79cc5c647f476c98101f9e48b22b9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067), d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473)
I20240503 14:12:04.290197 24528 raft_consensus.cc:504] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:04.291325 24074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac35b944198044e1af026f24d35e0b7b" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e"
I20240503 14:12:04.291591 24074 raft_consensus.cc:3037] T ac35b944198044e1af026f24d35e0b7b P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.292604 24528 leader_election.cc:288] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 election: Requested vote from peers 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675), ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067)
I20240503 14:12:04.294559 24074 raft_consensus.cc:2454] T ac35b944198044e1af026f24d35e0b7b P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 1.
I20240503 14:12:04.294754 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac35b944198044e1af026f24d35e0b7b" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74"
I20240503 14:12:04.294941 24337 raft_consensus.cc:3037] T ac35b944198044e1af026f24d35e0b7b P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.295025 23749 leader_election.cc:302] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, d4eea540f3c241c1ba6290ca53769163; no voters:
I20240503 14:12:04.295349 24528 raft_consensus.cc:2781] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:04.296290 23812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac35b944198044e1af026f24d35e0b7b" candidate_uuid: "1e79cc5c647f476c98101f9e48b22b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163" is_pre_election: true
I20240503 14:12:04.297914 24337 raft_consensus.cc:2454] T ac35b944198044e1af026f24d35e0b7b P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 1.
I20240503 14:12:04.303987 24528 raft_consensus.cc:686] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [term 1 LEADER]: Becoming Leader. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Running, Role: LEADER
I20240503 14:12:04.304530 24528 consensus_queue.cc:237] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:04.305140 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac35b944198044e1af026f24d35e0b7b" candidate_uuid: "1e79cc5c647f476c98101f9e48b22b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74" is_pre_election: true
I20240503 14:12:04.305449 24337 raft_consensus.cc:2379] T ac35b944198044e1af026f24d35e0b7b P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e79cc5c647f476c98101f9e48b22b9e in current term 1: Already voted for candidate d4eea540f3c241c1ba6290ca53769163 in this term.
I20240503 14:12:04.307556 24011 leader_election.cc:302] T ac35b944198044e1af026f24d35e0b7b P 1e79cc5c647f476c98101f9e48b22b9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e; no voters: d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74
I20240503 14:12:04.308013 24529 raft_consensus.cc:2726] T ac35b944198044e1af026f24d35e0b7b P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:04.308789 23536 catalog_manager.cc:5484] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4eea540f3c241c1ba6290ca53769163 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "d4eea540f3c241c1ba6290ca53769163" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:04.330421 24530 consensus_queue.cc:1035] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:04.335808 24552 consensus_queue.cc:1035] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:04.355484 24425 consensus_queue.cc:1035] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:04.360939 24448 consensus_queue.cc:1035] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:04.739432 24528 consensus_queue.cc:1035] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:04.745023 24547 consensus_queue.cc:1035] T ac35b944198044e1af026f24d35e0b7b P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:04.762784 24561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:04.763110 24561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:04.771274 24561 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 260.469 ms
time per row: 0.130234 ms
I20240503 14:12:05.186589 23535 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33990:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:05.196413 23660 tablet_service.cc:1467] Processing CreateTablet for tablet abf6fde8492d459cb00bc324e8dbc66b (DEFAULT_TABLE table=rebalance_test_table_2 [id=488bb1435ec14da7a0b7c6fa4e375e1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.196683 24185 tablet_service.cc:1467] Processing CreateTablet for tablet abf6fde8492d459cb00bc324e8dbc66b (DEFAULT_TABLE table=rebalance_test_table_2 [id=488bb1435ec14da7a0b7c6fa4e375e1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.196913 23660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abf6fde8492d459cb00bc324e8dbc66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.197098 24185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abf6fde8492d459cb00bc324e8dbc66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.200752 23924 tablet_service.cc:1467] Processing CreateTablet for tablet abf6fde8492d459cb00bc324e8dbc66b (DEFAULT_TABLE table=rebalance_test_table_2 [id=488bb1435ec14da7a0b7c6fa4e375e1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.201184 23924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abf6fde8492d459cb00bc324e8dbc66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.201771 24054 tablet_service.cc:1467] Processing CreateTablet for tablet efd2ded4212a49d38a35ef6e07650b74 (DEFAULT_TABLE table=rebalance_test_table_2 [id=488bb1435ec14da7a0b7c6fa4e375e1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.202046 23661 tablet_service.cc:1467] Processing CreateTablet for tablet 9814abfe435b4bdf9fb65d65d03bc1b9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=488bb1435ec14da7a0b7c6fa4e375e1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.202299 24054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efd2ded4212a49d38a35ef6e07650b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.202402 23661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9814abfe435b4bdf9fb65d65d03bc1b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.200825 23923 tablet_service.cc:1467] Processing CreateTablet for tablet 9814abfe435b4bdf9fb65d65d03bc1b9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=488bb1435ec14da7a0b7c6fa4e375e1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.204850 24186 tablet_service.cc:1467] Processing CreateTablet for tablet 9814abfe435b4bdf9fb65d65d03bc1b9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=488bb1435ec14da7a0b7c6fa4e375e1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.204998 23923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9814abfe435b4bdf9fb65d65d03bc1b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.205359 24186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9814abfe435b4bdf9fb65d65d03bc1b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.206370 23792 tablet_service.cc:1467] Processing CreateTablet for tablet efd2ded4212a49d38a35ef6e07650b74 (DEFAULT_TABLE table=rebalance_test_table_2 [id=488bb1435ec14da7a0b7c6fa4e375e1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.206768 23792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efd2ded4212a49d38a35ef6e07650b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.208750 24317 tablet_service.cc:1467] Processing CreateTablet for tablet efd2ded4212a49d38a35ef6e07650b74 (DEFAULT_TABLE table=rebalance_test_table_2 [id=488bb1435ec14da7a0b7c6fa4e375e1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.209282 24317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efd2ded4212a49d38a35ef6e07650b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.232892 24630 tablet_bootstrap.cc:492] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163: Bootstrap starting.
I20240503 14:12:05.239861 24633 tablet_bootstrap.cc:492] T abf6fde8492d459cb00bc324e8dbc66b P 8795a3c2a7014baf9126dea0a6ce1bb4: Bootstrap starting.
I20240503 14:12:05.241369 24632 tablet_bootstrap.cc:492] T efd2ded4212a49d38a35ef6e07650b74 P ff53511345114c4ab97a9a8decef3a74: Bootstrap starting.
I20240503 14:12:05.246150 24631 tablet_bootstrap.cc:492] T abf6fde8492d459cb00bc324e8dbc66b P c9a573f6faf64131b114af1e33c1b308: Bootstrap starting.
I20240503 14:12:05.247517 24634 tablet_bootstrap.cc:492] T efd2ded4212a49d38a35ef6e07650b74 P 1e79cc5c647f476c98101f9e48b22b9e: Bootstrap starting.
I20240503 14:12:05.247885 24629 tablet_bootstrap.cc:492] T 9814abfe435b4bdf9fb65d65d03bc1b9 P d40aeb06cfcc4c3a897b40290d12a15b: Bootstrap starting.
I20240503 14:12:05.249413 24629 tablet_bootstrap.cc:654] T 9814abfe435b4bdf9fb65d65d03bc1b9 P d40aeb06cfcc4c3a897b40290d12a15b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.249642 24634 tablet_bootstrap.cc:654] T efd2ded4212a49d38a35ef6e07650b74 P 1e79cc5c647f476c98101f9e48b22b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.251407 24633 tablet_bootstrap.cc:654] T abf6fde8492d459cb00bc324e8dbc66b P 8795a3c2a7014baf9126dea0a6ce1bb4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.253036 24632 tablet_bootstrap.cc:654] T efd2ded4212a49d38a35ef6e07650b74 P ff53511345114c4ab97a9a8decef3a74: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.254215 24629 tablet_bootstrap.cc:492] T 9814abfe435b4bdf9fb65d65d03bc1b9 P d40aeb06cfcc4c3a897b40290d12a15b: No bootstrap required, opened a new log
I20240503 14:12:05.255663 24629 ts_tablet_manager.cc:1367] T 9814abfe435b4bdf9fb65d65d03bc1b9 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240503 14:12:05.255952 24631 tablet_bootstrap.cc:654] T abf6fde8492d459cb00bc324e8dbc66b P c9a573f6faf64131b114af1e33c1b308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.256901 24630 tablet_bootstrap.cc:654] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.256907 24629 raft_consensus.cc:348] T 9814abfe435b4bdf9fb65d65d03bc1b9 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:05.257218 24629 raft_consensus.cc:374] T 9814abfe435b4bdf9fb65d65d03bc1b9 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.257357 24629 raft_consensus.cc:729] T 9814abfe435b4bdf9fb65d65d03bc1b9 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.257668 24629 consensus_queue.cc:260] T 9814abfe435b4bdf9fb65d65d03bc1b9 P d40aeb06cfcc4c3a897b40290d12a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:05.259855 24629 ts_tablet_manager.cc:1398] T 9814abfe435b4bdf9fb65d65d03bc1b9 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:05.260283 24629 tablet_bootstrap.cc:492] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b: Bootstrap starting.
I20240503 14:12:05.262529 24629 tablet_bootstrap.cc:654] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.264890 24633 tablet_bootstrap.cc:492] T abf6fde8492d459cb00bc324e8dbc66b P 8795a3c2a7014baf9126dea0a6ce1bb4: No bootstrap required, opened a new log
I20240503 14:12:05.265628 24633 ts_tablet_manager.cc:1367] T abf6fde8492d459cb00bc324e8dbc66b P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.010s
I20240503 14:12:05.266806 24633 raft_consensus.cc:348] T abf6fde8492d459cb00bc324e8dbc66b P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:05.267184 24633 raft_consensus.cc:374] T abf6fde8492d459cb00bc324e8dbc66b P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.267346 24633 raft_consensus.cc:729] T abf6fde8492d459cb00bc324e8dbc66b P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.267764 24633 consensus_queue.cc:260] T abf6fde8492d459cb00bc324e8dbc66b P 8795a3c2a7014baf9126dea0a6ce1bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:05.268638 24633 ts_tablet_manager.cc:1398] T abf6fde8492d459cb00bc324e8dbc66b P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:12:05.269603 24633 tablet_bootstrap.cc:492] T 9814abfe435b4bdf9fb65d65d03bc1b9 P 8795a3c2a7014baf9126dea0a6ce1bb4: Bootstrap starting.
I20240503 14:12:05.272331 24633 tablet_bootstrap.cc:654] T 9814abfe435b4bdf9fb65d65d03bc1b9 P 8795a3c2a7014baf9126dea0a6ce1bb4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.275045 24629 tablet_bootstrap.cc:492] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b: No bootstrap required, opened a new log
I20240503 14:12:05.275290 24629 ts_tablet_manager.cc:1367] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20240503 14:12:05.276384 24629 raft_consensus.cc:348] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:05.276752 24629 raft_consensus.cc:374] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.276886 24629 raft_consensus.cc:729] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.277256 24629 consensus_queue.cc:260] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:05.279150 24632 tablet_bootstrap.cc:492] T efd2ded4212a49d38a35ef6e07650b74 P ff53511345114c4ab97a9a8decef3a74: No bootstrap required, opened a new log
I20240503 14:12:05.279379 24632 ts_tablet_manager.cc:1367] T efd2ded4212a49d38a35ef6e07650b74 P ff53511345114c4ab97a9a8decef3a74: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.007s
I20240503 14:12:05.280143 24629 ts_tablet_manager.cc:1398] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:05.280212 24631 tablet_bootstrap.cc:492] T abf6fde8492d459cb00bc324e8dbc66b P c9a573f6faf64131b114af1e33c1b308: No bootstrap required, opened a new log
I20240503 14:12:05.280472 24631 ts_tablet_manager.cc:1367] T abf6fde8492d459cb00bc324e8dbc66b P c9a573f6faf64131b114af1e33c1b308: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.007s
I20240503 14:12:05.280450 24632 raft_consensus.cc:348] T efd2ded4212a49d38a35ef6e07650b74 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:05.280825 24632 raft_consensus.cc:374] T efd2ded4212a49d38a35ef6e07650b74 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.281018 24632 raft_consensus.cc:729] T efd2ded4212a49d38a35ef6e07650b74 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.281389 24632 consensus_queue.cc:260] T efd2ded4212a49d38a35ef6e07650b74 P ff53511345114c4ab97a9a8decef3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:05.281590 24631 raft_consensus.cc:348] T abf6fde8492d459cb00bc324e8dbc66b P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:05.281862 24631 raft_consensus.cc:374] T abf6fde8492d459cb00bc324e8dbc66b P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.281967 24631 raft_consensus.cc:729] T abf6fde8492d459cb00bc324e8dbc66b P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.282243 24631 consensus_queue.cc:260] T abf6fde8492d459cb00bc324e8dbc66b P c9a573f6faf64131b114af1e33c1b308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:05.282990 24631 ts_tablet_manager.cc:1398] T abf6fde8492d459cb00bc324e8dbc66b P c9a573f6faf64131b114af1e33c1b308: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:05.284235 24632 ts_tablet_manager.cc:1398] T efd2ded4212a49d38a35ef6e07650b74 P ff53511345114c4ab97a9a8decef3a74: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:05.285313 24633 tablet_bootstrap.cc:492] T 9814abfe435b4bdf9fb65d65d03bc1b9 P 8795a3c2a7014baf9126dea0a6ce1bb4: No bootstrap required, opened a new log
I20240503 14:12:05.285547 24633 ts_tablet_manager.cc:1367] T 9814abfe435b4bdf9fb65d65d03bc1b9 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.001s
I20240503 14:12:05.286446 24633 raft_consensus.cc:348] T 9814abfe435b4bdf9fb65d65d03bc1b9 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:05.286690 24633 raft_consensus.cc:374] T 9814abfe435b4bdf9fb65d65d03bc1b9 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.286770 24633 raft_consensus.cc:729] T 9814abfe435b4bdf9fb65d65d03bc1b9 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.287016 24633 consensus_queue.cc:260] T 9814abfe435b4bdf9fb65d65d03bc1b9 P 8795a3c2a7014baf9126dea0a6ce1bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:05.287528 24633 ts_tablet_manager.cc:1398] T 9814abfe435b4bdf9fb65d65d03bc1b9 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:05.288010 24631 tablet_bootstrap.cc:492] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308: Bootstrap starting.
I20240503 14:12:05.289251 24630 tablet_bootstrap.cc:492] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163: No bootstrap required, opened a new log
I20240503 14:12:05.289420 24630 ts_tablet_manager.cc:1367] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163: Time spent bootstrapping tablet: real 0.057s user 0.006s sys 0.000s
I20240503 14:12:05.289883 24631 tablet_bootstrap.cc:654] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.293440 24634 tablet_bootstrap.cc:492] T efd2ded4212a49d38a35ef6e07650b74 P 1e79cc5c647f476c98101f9e48b22b9e: No bootstrap required, opened a new log
I20240503 14:12:05.293668 24634 ts_tablet_manager.cc:1367] T efd2ded4212a49d38a35ef6e07650b74 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.026s
I20240503 14:12:05.294064 24630 raft_consensus.cc:348] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:05.294366 24630 raft_consensus.cc:374] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.294502 24630 raft_consensus.cc:729] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.294739 24634 raft_consensus.cc:348] T efd2ded4212a49d38a35ef6e07650b74 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:05.294826 24630 consensus_queue.cc:260] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:05.295075 24634 raft_consensus.cc:374] T efd2ded4212a49d38a35ef6e07650b74 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.295284 24634 raft_consensus.cc:729] T efd2ded4212a49d38a35ef6e07650b74 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.295423 24630 ts_tablet_manager.cc:1398] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:05.295635 24634 consensus_queue.cc:260] T efd2ded4212a49d38a35ef6e07650b74 P 1e79cc5c647f476c98101f9e48b22b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:05.298449 24634 ts_tablet_manager.cc:1398] T efd2ded4212a49d38a35ef6e07650b74 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240503 14:12:05.298822 24631 tablet_bootstrap.cc:492] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308: No bootstrap required, opened a new log
I20240503 14:12:05.298992 24631 ts_tablet_manager.cc:1367] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240503 14:12:05.299695 24631 raft_consensus.cc:348] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:05.299898 24631 raft_consensus.cc:374] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.299962 24631 raft_consensus.cc:729] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.300222 24631 consensus_queue.cc:260] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:05.301785 24631 ts_tablet_manager.cc:1398] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:05.420387 24602 raft_consensus.cc:482] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:05.420593 24602 raft_consensus.cc:504] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:05.421404 24602 leader_election.cc:288] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067), 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675)
I20240503 14:12:05.422003 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd2ded4212a49d38a35ef6e07650b74" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74" is_pre_election: true
I20240503 14:12:05.422025 24074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd2ded4212a49d38a35ef6e07650b74" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e" is_pre_election: true
I20240503 14:12:05.422262 24074 raft_consensus.cc:2454] T efd2ded4212a49d38a35ef6e07650b74 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 0.
I20240503 14:12:05.422298 24337 raft_consensus.cc:2454] T efd2ded4212a49d38a35ef6e07650b74 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 0.
I20240503 14:12:05.422777 23747 leader_election.cc:302] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:05.423108 24602 raft_consensus.cc:2781] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:05.423236 24602 raft_consensus.cc:482] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:05.423355 24602 raft_consensus.cc:3037] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.425383 24602 raft_consensus.cc:504] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:05.425984 24602 leader_election.cc:288] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 election: Requested vote from peers ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067), 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675)
I20240503 14:12:05.426489 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd2ded4212a49d38a35ef6e07650b74" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74"
I20240503 14:12:05.426507 24074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd2ded4212a49d38a35ef6e07650b74" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e"
I20240503 14:12:05.426703 24337 raft_consensus.cc:3037] T efd2ded4212a49d38a35ef6e07650b74 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.426707 24074 raft_consensus.cc:3037] T efd2ded4212a49d38a35ef6e07650b74 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.428809 24337 raft_consensus.cc:2454] T efd2ded4212a49d38a35ef6e07650b74 P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 1.
I20240503 14:12:05.428893 24074 raft_consensus.cc:2454] T efd2ded4212a49d38a35ef6e07650b74 P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 1.
I20240503 14:12:05.429265 23747 leader_election.cc:302] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:05.429559 24602 raft_consensus.cc:2781] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:05.429716 24602 raft_consensus.cc:686] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [term 1 LEADER]: Becoming Leader. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Running, Role: LEADER
I20240503 14:12:05.429975 24602 consensus_queue.cc:237] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:05.432642 23535 catalog_manager.cc:5484] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4eea540f3c241c1ba6290ca53769163 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "d4eea540f3c241c1ba6290ca53769163" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:05.564316 24493 raft_consensus.cc:482] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:05.564512 24493 raft_consensus.cc:504] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:05.565176 24493 leader_election.cc:288] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175), c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475)
I20240503 14:12:05.565842 24206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf6fde8492d459cb00bc324e8dbc66b" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308" is_pre_election: true
I20240503 14:12:05.565842 23681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf6fde8492d459cb00bc324e8dbc66b" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" is_pre_election: true
I20240503 14:12:05.566083 24206 raft_consensus.cc:2454] T abf6fde8492d459cb00bc324e8dbc66b P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 0.
I20240503 14:12:05.566083 23681 raft_consensus.cc:2454] T abf6fde8492d459cb00bc324e8dbc66b P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 0.
I20240503 14:12:05.566577 23878 leader_election.cc:302] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:05.566905 24493 raft_consensus.cc:2781] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:05.567001 24493 raft_consensus.cc:482] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:05.567090 24493 raft_consensus.cc:3037] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.569036 24493 raft_consensus.cc:504] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:05.569527 24493 leader_election.cc:288] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 election: Requested vote from peers 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175), c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475)
I20240503 14:12:05.569974 23681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf6fde8492d459cb00bc324e8dbc66b" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4"
I20240503 14:12:05.570024 24206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf6fde8492d459cb00bc324e8dbc66b" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308"
I20240503 14:12:05.570140 23681 raft_consensus.cc:3037] T abf6fde8492d459cb00bc324e8dbc66b P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.570228 24206 raft_consensus.cc:3037] T abf6fde8492d459cb00bc324e8dbc66b P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.571916 24206 raft_consensus.cc:2454] T abf6fde8492d459cb00bc324e8dbc66b P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 1.
I20240503 14:12:05.571920 23681 raft_consensus.cc:2454] T abf6fde8492d459cb00bc324e8dbc66b P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 1.
I20240503 14:12:05.572309 23878 leader_election.cc:302] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:05.572641 24493 raft_consensus.cc:2781] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:05.572986 24493 raft_consensus.cc:686] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [term 1 LEADER]: Becoming Leader. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Running, Role: LEADER
I20240503 14:12:05.573300 24493 consensus_queue.cc:237] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:05.576192 23535 catalog_manager.cc:5484] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b reported cstate change: term changed from 0 to 1, leader changed from <none> to d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67). New cstate: current_term: 1 leader_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:05.690965 24586 raft_consensus.cc:482] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:05.691152 24586 raft_consensus.cc:504] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:05.691845 24586 leader_election.cc:288] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697), 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175)
I20240503 14:12:05.692338 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9814abfe435b4bdf9fb65d65d03bc1b9" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" is_pre_election: true
I20240503 14:12:05.692445 23681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9814abfe435b4bdf9fb65d65d03bc1b9" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" is_pre_election: true
I20240503 14:12:05.692574 23944 raft_consensus.cc:2454] T 9814abfe435b4bdf9fb65d65d03bc1b9 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 0.
I20240503 14:12:05.692699 23681 raft_consensus.cc:2454] T 9814abfe435b4bdf9fb65d65d03bc1b9 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 0.
I20240503 14:12:05.693035 24139 leader_election.cc:302] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:05.693308 24586 raft_consensus.cc:2781] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:05.693425 24586 raft_consensus.cc:482] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:05.693506 24586 raft_consensus.cc:3037] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.696362 24586 raft_consensus.cc:504] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:05.697052 24586 leader_election.cc:288] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 election: Requested vote from peers d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697), 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175)
I20240503 14:12:05.697535 23681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9814abfe435b4bdf9fb65d65d03bc1b9" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4"
I20240503 14:12:05.697566 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9814abfe435b4bdf9fb65d65d03bc1b9" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b"
I20240503 14:12:05.697749 23681 raft_consensus.cc:3037] T 9814abfe435b4bdf9fb65d65d03bc1b9 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.697784 23944 raft_consensus.cc:3037] T 9814abfe435b4bdf9fb65d65d03bc1b9 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.699537 23681 raft_consensus.cc:2454] T 9814abfe435b4bdf9fb65d65d03bc1b9 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 1.
I20240503 14:12:05.699944 24140 leader_election.cc:302] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8795a3c2a7014baf9126dea0a6ce1bb4, c9a573f6faf64131b114af1e33c1b308; no voters:
I20240503 14:12:05.700250 24586 raft_consensus.cc:2781] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:05.700536 23944 raft_consensus.cc:2454] T 9814abfe435b4bdf9fb65d65d03bc1b9 P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 1.
I20240503 14:12:05.700654 24586 raft_consensus.cc:686] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [term 1 LEADER]: Becoming Leader. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Running, Role: LEADER
I20240503 14:12:05.700996 24586 consensus_queue.cc:237] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:05.703655 23535 catalog_manager.cc:5484] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9a573f6faf64131b114af1e33c1b308 (127.22.242.69). New cstate: current_term: 1 leader_uuid: "c9a573f6faf64131b114af1e33c1b308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:05.810335 24602 consensus_queue.cc:1035] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:05.817111 24597 consensus_queue.cc:1035] T efd2ded4212a49d38a35ef6e07650b74 P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:06.054842 24493 consensus_queue.cc:1035] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:06.060511 24647 consensus_queue.cc:1035] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:06.092497 24586 consensus_queue.cc:1035] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:06.097976 24649 consensus_queue.cc:1035] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:06.125885 24651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:06.126243 24651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:06.133936 24651 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 329.733 ms
time per row: 0.164867 ms
I20240503 14:12:06.621238 23540 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33990:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:06.629355 24054 tablet_service.cc:1467] Processing CreateTablet for tablet 944a3478fc06494384f0956bf575b8e6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0279efb3cfd9405daa01255bdbd7b367]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.629463 23792 tablet_service.cc:1467] Processing CreateTablet for tablet 944a3478fc06494384f0956bf575b8e6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0279efb3cfd9405daa01255bdbd7b367]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.629841 24054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944a3478fc06494384f0956bf575b8e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.629884 23792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944a3478fc06494384f0956bf575b8e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.630465 24316 tablet_service.cc:1467] Processing CreateTablet for tablet 5c55248037ca497bbc153b5933d4230e (DEFAULT_TABLE table=rebalance_test_table_3 [id=0279efb3cfd9405daa01255bdbd7b367]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.630852 24316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c55248037ca497bbc153b5933d4230e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.632625 23660 tablet_service.cc:1467] Processing CreateTablet for tablet 666732f9b38f4a20a667bbb9880ece02 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0279efb3cfd9405daa01255bdbd7b367]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.632761 23793 tablet_service.cc:1467] Processing CreateTablet for tablet 5c55248037ca497bbc153b5933d4230e (DEFAULT_TABLE table=rebalance_test_table_3 [id=0279efb3cfd9405daa01255bdbd7b367]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.633050 23660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 666732f9b38f4a20a667bbb9880ece02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.633114 23793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c55248037ca497bbc153b5933d4230e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.637802 23923 tablet_service.cc:1467] Processing CreateTablet for tablet 666732f9b38f4a20a667bbb9880ece02 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0279efb3cfd9405daa01255bdbd7b367]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.637867 24185 tablet_service.cc:1467] Processing CreateTablet for tablet 666732f9b38f4a20a667bbb9880ece02 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0279efb3cfd9405daa01255bdbd7b367]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.637858 24055 tablet_service.cc:1467] Processing CreateTablet for tablet 5c55248037ca497bbc153b5933d4230e (DEFAULT_TABLE table=rebalance_test_table_3 [id=0279efb3cfd9405daa01255bdbd7b367]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.638239 23923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 666732f9b38f4a20a667bbb9880ece02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.638260 24055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c55248037ca497bbc153b5933d4230e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.638267 24185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 666732f9b38f4a20a667bbb9880ece02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.644387 24317 tablet_service.cc:1467] Processing CreateTablet for tablet 944a3478fc06494384f0956bf575b8e6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0279efb3cfd9405daa01255bdbd7b367]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.644915 24317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944a3478fc06494384f0956bf575b8e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.653151 24730 tablet_bootstrap.cc:492] T 666732f9b38f4a20a667bbb9880ece02 P 8795a3c2a7014baf9126dea0a6ce1bb4: Bootstrap starting.
I20240503 14:12:06.655499 24730 tablet_bootstrap.cc:654] T 666732f9b38f4a20a667bbb9880ece02 P 8795a3c2a7014baf9126dea0a6ce1bb4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:06.666299 24728 tablet_bootstrap.cc:492] T 944a3478fc06494384f0956bf575b8e6 P 1e79cc5c647f476c98101f9e48b22b9e: Bootstrap starting.
I20240503 14:12:06.667438 24733 tablet_bootstrap.cc:492] T 5c55248037ca497bbc153b5933d4230e P ff53511345114c4ab97a9a8decef3a74: Bootstrap starting.
I20240503 14:12:06.669821 24733 tablet_bootstrap.cc:654] T 5c55248037ca497bbc153b5933d4230e P ff53511345114c4ab97a9a8decef3a74: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:06.671797 24731 tablet_bootstrap.cc:492] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308: Bootstrap starting.
I20240503 14:12:06.673225 24728 tablet_bootstrap.cc:654] T 944a3478fc06494384f0956bf575b8e6 P 1e79cc5c647f476c98101f9e48b22b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:06.674095 24729 tablet_bootstrap.cc:492] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163: Bootstrap starting.
I20240503 14:12:06.674129 24731 tablet_bootstrap.cc:654] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:06.677085 24730 tablet_bootstrap.cc:492] T 666732f9b38f4a20a667bbb9880ece02 P 8795a3c2a7014baf9126dea0a6ce1bb4: No bootstrap required, opened a new log
I20240503 14:12:06.677366 24730 ts_tablet_manager.cc:1367] T 666732f9b38f4a20a667bbb9880ece02 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.004s
I20240503 14:12:06.677408 24728 tablet_bootstrap.cc:492] T 944a3478fc06494384f0956bf575b8e6 P 1e79cc5c647f476c98101f9e48b22b9e: No bootstrap required, opened a new log
I20240503 14:12:06.677400 24734 tablet_bootstrap.cc:492] T 666732f9b38f4a20a667bbb9880ece02 P d40aeb06cfcc4c3a897b40290d12a15b: Bootstrap starting.
I20240503 14:12:06.677544 24728 ts_tablet_manager.cc:1367] T 944a3478fc06494384f0956bf575b8e6 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 14:12:06.678398 24728 raft_consensus.cc:348] T 944a3478fc06494384f0956bf575b8e6 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:06.678716 24728 raft_consensus.cc:374] T 944a3478fc06494384f0956bf575b8e6 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:06.678555 24730 raft_consensus.cc:348] T 666732f9b38f4a20a667bbb9880ece02 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:06.678813 24730 raft_consensus.cc:374] T 666732f9b38f4a20a667bbb9880ece02 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:06.678891 24728 raft_consensus.cc:729] T 944a3478fc06494384f0956bf575b8e6 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Initialized, Role: FOLLOWER
I20240503 14:12:06.679137 24734 tablet_bootstrap.cc:654] T 666732f9b38f4a20a667bbb9880ece02 P d40aeb06cfcc4c3a897b40290d12a15b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:06.679210 24728 consensus_queue.cc:260] T 944a3478fc06494384f0956bf575b8e6 P 1e79cc5c647f476c98101f9e48b22b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:06.679471 24730 raft_consensus.cc:729] T 666732f9b38f4a20a667bbb9880ece02 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Initialized, Role: FOLLOWER
I20240503 14:12:06.679342 24729 tablet_bootstrap.cc:654] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:06.680699 24730 consensus_queue.cc:260] T 666732f9b38f4a20a667bbb9880ece02 P 8795a3c2a7014baf9126dea0a6ce1bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:06.682596 24733 tablet_bootstrap.cc:492] T 5c55248037ca497bbc153b5933d4230e P ff53511345114c4ab97a9a8decef3a74: No bootstrap required, opened a new log
I20240503 14:12:06.682842 24733 ts_tablet_manager.cc:1367] T 5c55248037ca497bbc153b5933d4230e P ff53511345114c4ab97a9a8decef3a74: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240503 14:12:06.683636 24731 tablet_bootstrap.cc:492] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308: No bootstrap required, opened a new log
I20240503 14:12:06.683858 24731 ts_tablet_manager.cc:1367] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240503 14:12:06.684342 24733 raft_consensus.cc:348] T 5c55248037ca497bbc153b5933d4230e P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:06.684679 24733 raft_consensus.cc:374] T 5c55248037ca497bbc153b5933d4230e P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:06.684826 24733 raft_consensus.cc:729] T 5c55248037ca497bbc153b5933d4230e P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Initialized, Role: FOLLOWER
I20240503 14:12:06.684965 24731 raft_consensus.cc:348] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:06.685259 24731 raft_consensus.cc:374] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:06.685374 24731 raft_consensus.cc:729] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Initialized, Role: FOLLOWER
I20240503 14:12:06.685317 24733 consensus_queue.cc:260] T 5c55248037ca497bbc153b5933d4230e P ff53511345114c4ab97a9a8decef3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:06.685772 24731 consensus_queue.cc:260] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:06.686533 24731 ts_tablet_manager.cc:1398] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:06.687436 24728 ts_tablet_manager.cc:1398] T 944a3478fc06494384f0956bf575b8e6 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent starting tablet: real 0.010s user 0.004s sys 0.005s
I20240503 14:12:06.687742 24728 tablet_bootstrap.cc:492] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e: Bootstrap starting.
I20240503 14:12:06.689652 24728 tablet_bootstrap.cc:654] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:06.695871 24734 tablet_bootstrap.cc:492] T 666732f9b38f4a20a667bbb9880ece02 P d40aeb06cfcc4c3a897b40290d12a15b: No bootstrap required, opened a new log
I20240503 14:12:06.696072 24734 ts_tablet_manager.cc:1367] T 666732f9b38f4a20a667bbb9880ece02 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.003s
I20240503 14:12:06.697187 24734 raft_consensus.cc:348] T 666732f9b38f4a20a667bbb9880ece02 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:06.697507 24734 raft_consensus.cc:374] T 666732f9b38f4a20a667bbb9880ece02 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:06.697652 24734 raft_consensus.cc:729] T 666732f9b38f4a20a667bbb9880ece02 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Initialized, Role: FOLLOWER
I20240503 14:12:06.697997 24734 consensus_queue.cc:260] T 666732f9b38f4a20a667bbb9880ece02 P d40aeb06cfcc4c3a897b40290d12a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:06.699030 24734 ts_tablet_manager.cc:1398] T 666732f9b38f4a20a667bbb9880ece02 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:06.700759 24730 ts_tablet_manager.cc:1398] T 666732f9b38f4a20a667bbb9880ece02 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240503 14:12:06.701287 24733 ts_tablet_manager.cc:1398] T 5c55248037ca497bbc153b5933d4230e P ff53511345114c4ab97a9a8decef3a74: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240503 14:12:06.701576 24733 tablet_bootstrap.cc:492] T 944a3478fc06494384f0956bf575b8e6 P ff53511345114c4ab97a9a8decef3a74: Bootstrap starting.
I20240503 14:12:06.703316 24733 tablet_bootstrap.cc:654] T 944a3478fc06494384f0956bf575b8e6 P ff53511345114c4ab97a9a8decef3a74: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:06.703629 24729 tablet_bootstrap.cc:492] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163: No bootstrap required, opened a new log
I20240503 14:12:06.703871 24729 ts_tablet_manager.cc:1367] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.019s
I20240503 14:12:06.704994 24729 raft_consensus.cc:348] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:06.705297 24729 raft_consensus.cc:374] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:06.705454 24729 raft_consensus.cc:729] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Initialized, Role: FOLLOWER
I20240503 14:12:06.705777 24729 consensus_queue.cc:260] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:06.706521 24729 ts_tablet_manager.cc:1398] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:06.708794 24728 tablet_bootstrap.cc:492] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e: No bootstrap required, opened a new log
I20240503 14:12:06.708855 24733 tablet_bootstrap.cc:492] T 944a3478fc06494384f0956bf575b8e6 P ff53511345114c4ab97a9a8decef3a74: No bootstrap required, opened a new log
I20240503 14:12:06.709074 24728 ts_tablet_manager.cc:1367] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.013s
I20240503 14:12:06.709122 24733 ts_tablet_manager.cc:1367] T 944a3478fc06494384f0956bf575b8e6 P ff53511345114c4ab97a9a8decef3a74: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240503 14:12:06.710004 24733 raft_consensus.cc:348] T 944a3478fc06494384f0956bf575b8e6 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:06.710356 24733 raft_consensus.cc:374] T 944a3478fc06494384f0956bf575b8e6 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:06.710487 24733 raft_consensus.cc:729] T 944a3478fc06494384f0956bf575b8e6 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Initialized, Role: FOLLOWER
I20240503 14:12:06.710794 24733 consensus_queue.cc:260] T 944a3478fc06494384f0956bf575b8e6 P ff53511345114c4ab97a9a8decef3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:06.711388 24733 ts_tablet_manager.cc:1398] T 944a3478fc06494384f0956bf575b8e6 P ff53511345114c4ab97a9a8decef3a74: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:06.712479 24729 tablet_bootstrap.cc:492] T 5c55248037ca497bbc153b5933d4230e P d4eea540f3c241c1ba6290ca53769163: Bootstrap starting.
I20240503 14:12:06.714417 24729 tablet_bootstrap.cc:654] T 5c55248037ca497bbc153b5933d4230e P d4eea540f3c241c1ba6290ca53769163: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:06.720458 24729 tablet_bootstrap.cc:492] T 5c55248037ca497bbc153b5933d4230e P d4eea540f3c241c1ba6290ca53769163: No bootstrap required, opened a new log
I20240503 14:12:06.720664 24729 ts_tablet_manager.cc:1367] T 5c55248037ca497bbc153b5933d4230e P d4eea540f3c241c1ba6290ca53769163: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20240503 14:12:06.721105 24728 raft_consensus.cc:348] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:06.721421 24728 raft_consensus.cc:374] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:06.721462 24729 raft_consensus.cc:348] T 5c55248037ca497bbc153b5933d4230e P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:06.721577 24728 raft_consensus.cc:729] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Initialized, Role: FOLLOWER
I20240503 14:12:06.721706 24729 raft_consensus.cc:374] T 5c55248037ca497bbc153b5933d4230e P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:06.721798 24729 raft_consensus.cc:729] T 5c55248037ca497bbc153b5933d4230e P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Initialized, Role: FOLLOWER
I20240503 14:12:06.722031 24728 consensus_queue.cc:260] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:06.722072 24729 consensus_queue.cc:260] T 5c55248037ca497bbc153b5933d4230e P d4eea540f3c241c1ba6290ca53769163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:06.725958 24729 ts_tablet_manager.cc:1398] T 5c55248037ca497bbc153b5933d4230e P d4eea540f3c241c1ba6290ca53769163: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 14:12:06.727162 24728 ts_tablet_manager.cc:1398] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240503 14:12:06.739786 24649 raft_consensus.cc:482] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:06.740017 24649 raft_consensus.cc:504] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:06.740804 24649 leader_election.cc:288] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175), d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697)
I20240503 14:12:06.741308 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "666732f9b38f4a20a667bbb9880ece02" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" is_pre_election: true
I20240503 14:12:06.741333 23681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "666732f9b38f4a20a667bbb9880ece02" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" is_pre_election: true
I20240503 14:12:06.741524 23681 raft_consensus.cc:2454] T 666732f9b38f4a20a667bbb9880ece02 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 0.
I20240503 14:12:06.741528 23944 raft_consensus.cc:2454] T 666732f9b38f4a20a667bbb9880ece02 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 0.
I20240503 14:12:06.741993 24140 leader_election.cc:302] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8795a3c2a7014baf9126dea0a6ce1bb4, c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:06.742290 24649 raft_consensus.cc:2781] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:06.742390 24649 raft_consensus.cc:482] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:06.742512 24649 raft_consensus.cc:3037] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:06.744590 24649 raft_consensus.cc:504] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:06.745200 24649 leader_election.cc:288] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 election: Requested vote from peers 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175), d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697)
I20240503 14:12:06.745651 23681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "666732f9b38f4a20a667bbb9880ece02" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4"
I20240503 14:12:06.745666 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "666732f9b38f4a20a667bbb9880ece02" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b"
I20240503 14:12:06.745839 23681 raft_consensus.cc:3037] T 666732f9b38f4a20a667bbb9880ece02 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:06.745875 23944 raft_consensus.cc:3037] T 666732f9b38f4a20a667bbb9880ece02 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:06.748243 23681 raft_consensus.cc:2454] T 666732f9b38f4a20a667bbb9880ece02 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 1.
I20240503 14:12:06.748246 23944 raft_consensus.cc:2454] T 666732f9b38f4a20a667bbb9880ece02 P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 1.
I20240503 14:12:06.748674 24140 leader_election.cc:302] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8795a3c2a7014baf9126dea0a6ce1bb4, c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:06.748953 24649 raft_consensus.cc:2781] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:06.749114 24649 raft_consensus.cc:686] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [term 1 LEADER]: Becoming Leader. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Running, Role: LEADER
I20240503 14:12:06.749421 24649 consensus_queue.cc:237] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:06.751960 23531 catalog_manager.cc:5484] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9a573f6faf64131b114af1e33c1b308 (127.22.242.69). New cstate: current_term: 1 leader_uuid: "c9a573f6faf64131b114af1e33c1b308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:06.756876 24708 raft_consensus.cc:482] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:06.757035 24708 raft_consensus.cc:504] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:06.757666 24708 leader_election.cc:288] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675), ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067)
I20240503 14:12:06.758160 24075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944a3478fc06494384f0956bf575b8e6" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e" is_pre_election: true
I20240503 14:12:06.758198 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944a3478fc06494384f0956bf575b8e6" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74" is_pre_election: true
I20240503 14:12:06.758387 24075 raft_consensus.cc:2454] T 944a3478fc06494384f0956bf575b8e6 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 0.
I20240503 14:12:06.758390 24337 raft_consensus.cc:2454] T 944a3478fc06494384f0956bf575b8e6 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 0.
I20240503 14:12:06.758791 23749 leader_election.cc:302] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, d4eea540f3c241c1ba6290ca53769163; no voters:
I20240503 14:12:06.759039 24708 raft_consensus.cc:2781] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:06.759158 24708 raft_consensus.cc:482] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:06.759246 24708 raft_consensus.cc:3037] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:06.761130 24708 raft_consensus.cc:504] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:06.761626 24708 leader_election.cc:288] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 election: Requested vote from peers 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675), ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067)
I20240503 14:12:06.761973 24075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944a3478fc06494384f0956bf575b8e6" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e"
I20240503 14:12:06.762156 24075 raft_consensus.cc:3037] T 944a3478fc06494384f0956bf575b8e6 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:06.762161 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944a3478fc06494384f0956bf575b8e6" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74"
I20240503 14:12:06.762356 24337 raft_consensus.cc:3037] T 944a3478fc06494384f0956bf575b8e6 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:06.763926 24075 raft_consensus.cc:2454] T 944a3478fc06494384f0956bf575b8e6 P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 1.
I20240503 14:12:06.764148 24337 raft_consensus.cc:2454] T 944a3478fc06494384f0956bf575b8e6 P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 1.
I20240503 14:12:06.764341 23749 leader_election.cc:302] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, d4eea540f3c241c1ba6290ca53769163; no voters:
I20240503 14:12:06.764623 24708 raft_consensus.cc:2781] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:06.764768 24708 raft_consensus.cc:686] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [term 1 LEADER]: Becoming Leader. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Running, Role: LEADER
I20240503 14:12:06.765045 24708 consensus_queue.cc:237] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:06.767632 23531 catalog_manager.cc:5484] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4eea540f3c241c1ba6290ca53769163 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "d4eea540f3c241c1ba6290ca53769163" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:07.036743 24738 raft_consensus.cc:482] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:07.036924 24738 raft_consensus.cc:504] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:07.037575 24738 leader_election.cc:288] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067), d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473)
I20240503 14:12:07.038086 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c55248037ca497bbc153b5933d4230e" candidate_uuid: "1e79cc5c647f476c98101f9e48b22b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74" is_pre_election: true
I20240503 14:12:07.038164 23813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c55248037ca497bbc153b5933d4230e" candidate_uuid: "1e79cc5c647f476c98101f9e48b22b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163" is_pre_election: true
I20240503 14:12:07.038372 23813 raft_consensus.cc:2454] T 5c55248037ca497bbc153b5933d4230e P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e79cc5c647f476c98101f9e48b22b9e in term 0.
I20240503 14:12:07.038357 24337 raft_consensus.cc:2454] T 5c55248037ca497bbc153b5933d4230e P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e79cc5c647f476c98101f9e48b22b9e in term 0.
I20240503 14:12:07.038813 24009 leader_election.cc:302] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:07.039089 24738 raft_consensus.cc:2781] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:07.039194 24738 raft_consensus.cc:482] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:07.039278 24738 raft_consensus.cc:3037] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.041399 24738 raft_consensus.cc:504] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:07.041963 24738 leader_election.cc:288] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [CANDIDATE]: Term 1 election: Requested vote from peers ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067), d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473)
I20240503 14:12:07.042388 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c55248037ca497bbc153b5933d4230e" candidate_uuid: "1e79cc5c647f476c98101f9e48b22b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74"
I20240503 14:12:07.042412 23813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c55248037ca497bbc153b5933d4230e" candidate_uuid: "1e79cc5c647f476c98101f9e48b22b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163"
I20240503 14:12:07.042580 24337 raft_consensus.cc:3037] T 5c55248037ca497bbc153b5933d4230e P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.042598 23813 raft_consensus.cc:3037] T 5c55248037ca497bbc153b5933d4230e P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.044414 24337 raft_consensus.cc:2454] T 5c55248037ca497bbc153b5933d4230e P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e79cc5c647f476c98101f9e48b22b9e in term 1.
I20240503 14:12:07.044396 23813 raft_consensus.cc:2454] T 5c55248037ca497bbc153b5933d4230e P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e79cc5c647f476c98101f9e48b22b9e in term 1.
I20240503 14:12:07.044772 24009 leader_election.cc:302] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:07.045142 24738 raft_consensus.cc:2781] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:07.045488 24738 raft_consensus.cc:686] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [term 1 LEADER]: Becoming Leader. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Running, Role: LEADER
I20240503 14:12:07.045939 24738 consensus_queue.cc:237] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:07.049185 23531 catalog_manager.cc:5484] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68). New cstate: current_term: 1 leader_uuid: "1e79cc5c647f476c98101f9e48b22b9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:07.160707 24708 consensus_queue.cc:1035] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:07.166246 24708 consensus_queue.cc:1035] T 944a3478fc06494384f0956bf575b8e6 P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:07.284888 24649 consensus_queue.cc:1035] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:07.298032 24757 consensus_queue.cc:1035] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:07.437448 24738 consensus_queue.cc:1035] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:07.444383 24746 consensus_queue.cc:1035] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:07.476886 24748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:07.477339 24748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:07.486163 24748 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 303.369 ms
time per row: 0.151684 ms
I20240503 14:12:07.941375 23531 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33990:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:07.950569 23923 tablet_service.cc:1467] Processing CreateTablet for tablet 916b8dad2a52497799b76127ce499a2f (DEFAULT_TABLE table=rebalance_test_table_4 [id=c73d9911a53f447dbc232f4b8940c322]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.951057 23923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916b8dad2a52497799b76127ce499a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.950937 24185 tablet_service.cc:1467] Processing CreateTablet for tablet 916b8dad2a52497799b76127ce499a2f (DEFAULT_TABLE table=rebalance_test_table_4 [id=c73d9911a53f447dbc232f4b8940c322]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.951514 24185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916b8dad2a52497799b76127ce499a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.951800 23660 tablet_service.cc:1467] Processing CreateTablet for tablet 916b8dad2a52497799b76127ce499a2f (DEFAULT_TABLE table=rebalance_test_table_4 [id=c73d9911a53f447dbc232f4b8940c322]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.952227 23660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916b8dad2a52497799b76127ce499a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.952847 23924 tablet_service.cc:1467] Processing CreateTablet for tablet 16e304b5783546aea642a1525767007e (DEFAULT_TABLE table=rebalance_test_table_4 [id=c73d9911a53f447dbc232f4b8940c322]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.953100 23924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16e304b5783546aea642a1525767007e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.956729 24316 tablet_service.cc:1467] Processing CreateTablet for tablet d70414f531ac463c9e89e2236d94250c (DEFAULT_TABLE table=rebalance_test_table_4 [id=c73d9911a53f447dbc232f4b8940c322]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.957176 24316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d70414f531ac463c9e89e2236d94250c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.960673 23661 tablet_service.cc:1467] Processing CreateTablet for tablet 16e304b5783546aea642a1525767007e (DEFAULT_TABLE table=rebalance_test_table_4 [id=c73d9911a53f447dbc232f4b8940c322]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.961211 23661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16e304b5783546aea642a1525767007e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.965621 24186 tablet_service.cc:1467] Processing CreateTablet for tablet 16e304b5783546aea642a1525767007e (DEFAULT_TABLE table=rebalance_test_table_4 [id=c73d9911a53f447dbc232f4b8940c322]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.966153 24186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16e304b5783546aea642a1525767007e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.969727 24054 tablet_service.cc:1467] Processing CreateTablet for tablet d70414f531ac463c9e89e2236d94250c (DEFAULT_TABLE table=rebalance_test_table_4 [id=c73d9911a53f447dbc232f4b8940c322]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.969905 23792 tablet_service.cc:1467] Processing CreateTablet for tablet d70414f531ac463c9e89e2236d94250c (DEFAULT_TABLE table=rebalance_test_table_4 [id=c73d9911a53f447dbc232f4b8940c322]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.970278 24054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d70414f531ac463c9e89e2236d94250c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.970366 23792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d70414f531ac463c9e89e2236d94250c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.972640 24820 tablet_bootstrap.cc:492] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4: Bootstrap starting.
I20240503 14:12:07.974458 24820 tablet_bootstrap.cc:654] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.987159 24824 tablet_bootstrap.cc:492] T 16e304b5783546aea642a1525767007e P c9a573f6faf64131b114af1e33c1b308: Bootstrap starting.
I20240503 14:12:07.989684 24824 tablet_bootstrap.cc:654] T 16e304b5783546aea642a1525767007e P c9a573f6faf64131b114af1e33c1b308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.989879 24822 tablet_bootstrap.cc:492] T d70414f531ac463c9e89e2236d94250c P d4eea540f3c241c1ba6290ca53769163: Bootstrap starting.
I20240503 14:12:07.992233 24822 tablet_bootstrap.cc:654] T d70414f531ac463c9e89e2236d94250c P d4eea540f3c241c1ba6290ca53769163: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.992476 24819 tablet_bootstrap.cc:492] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b: Bootstrap starting.
I20240503 14:12:07.992853 24825 tablet_bootstrap.cc:492] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74: Bootstrap starting.
I20240503 14:12:07.994544 24819 tablet_bootstrap.cc:654] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.996851 24825 tablet_bootstrap.cc:654] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.001292 24820 tablet_bootstrap.cc:492] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4: No bootstrap required, opened a new log
I20240503 14:12:08.001642 24820 ts_tablet_manager.cc:1367] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.015s
I20240503 14:12:08.002318 24822 tablet_bootstrap.cc:492] T d70414f531ac463c9e89e2236d94250c P d4eea540f3c241c1ba6290ca53769163: No bootstrap required, opened a new log
I20240503 14:12:08.002658 24822 ts_tablet_manager.cc:1367] T d70414f531ac463c9e89e2236d94250c P d4eea540f3c241c1ba6290ca53769163: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240503 14:12:08.002821 24820 raft_consensus.cc:348] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:08.003119 24820 raft_consensus.cc:374] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.003247 24820 raft_consensus.cc:729] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.003554 24820 consensus_queue.cc:260] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:08.004324 24820 ts_tablet_manager.cc:1398] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:08.004654 24820 tablet_bootstrap.cc:492] T 16e304b5783546aea642a1525767007e P 8795a3c2a7014baf9126dea0a6ce1bb4: Bootstrap starting.
I20240503 14:12:08.004595 24822 raft_consensus.cc:348] T d70414f531ac463c9e89e2236d94250c P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:08.006726 24820 tablet_bootstrap.cc:654] T 16e304b5783546aea642a1525767007e P 8795a3c2a7014baf9126dea0a6ce1bb4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.006825 24822 raft_consensus.cc:374] T d70414f531ac463c9e89e2236d94250c P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.006978 24822 raft_consensus.cc:729] T d70414f531ac463c9e89e2236d94250c P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.007274 24825 tablet_bootstrap.cc:492] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74: No bootstrap required, opened a new log
I20240503 14:12:08.008225 24823 tablet_bootstrap.cc:492] T d70414f531ac463c9e89e2236d94250c P 1e79cc5c647f476c98101f9e48b22b9e: Bootstrap starting.
I20240503 14:12:08.009809 24824 tablet_bootstrap.cc:492] T 16e304b5783546aea642a1525767007e P c9a573f6faf64131b114af1e33c1b308: No bootstrap required, opened a new log
I20240503 14:12:08.009963 24824 ts_tablet_manager.cc:1367] T 16e304b5783546aea642a1525767007e P c9a573f6faf64131b114af1e33c1b308: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20240503 14:12:08.010531 24823 tablet_bootstrap.cc:654] T d70414f531ac463c9e89e2236d94250c P 1e79cc5c647f476c98101f9e48b22b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.010846 24824 raft_consensus.cc:348] T 16e304b5783546aea642a1525767007e P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:08.011054 24824 raft_consensus.cc:374] T 16e304b5783546aea642a1525767007e P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.011142 24824 raft_consensus.cc:729] T 16e304b5783546aea642a1525767007e P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.011368 24824 consensus_queue.cc:260] T 16e304b5783546aea642a1525767007e P c9a573f6faf64131b114af1e33c1b308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:08.012887 24820 tablet_bootstrap.cc:492] T 16e304b5783546aea642a1525767007e P 8795a3c2a7014baf9126dea0a6ce1bb4: No bootstrap required, opened a new log
I20240503 14:12:08.013141 24820 ts_tablet_manager.cc:1367] T 16e304b5783546aea642a1525767007e P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20240503 14:12:08.013831 24824 ts_tablet_manager.cc:1398] T 16e304b5783546aea642a1525767007e P c9a573f6faf64131b114af1e33c1b308: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:08.014279 24820 raft_consensus.cc:348] T 16e304b5783546aea642a1525767007e P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:08.014670 24820 raft_consensus.cc:374] T 16e304b5783546aea642a1525767007e P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.014824 24820 raft_consensus.cc:729] T 16e304b5783546aea642a1525767007e P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.015311 24820 consensus_queue.cc:260] T 16e304b5783546aea642a1525767007e P 8795a3c2a7014baf9126dea0a6ce1bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:08.016372 24820 ts_tablet_manager.cc:1398] T 16e304b5783546aea642a1525767007e P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:08.017560 24824 tablet_bootstrap.cc:492] T 916b8dad2a52497799b76127ce499a2f P c9a573f6faf64131b114af1e33c1b308: Bootstrap starting.
I20240503 14:12:08.019778 24824 tablet_bootstrap.cc:654] T 916b8dad2a52497799b76127ce499a2f P c9a573f6faf64131b114af1e33c1b308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.007622 24822 consensus_queue.cc:260] T d70414f531ac463c9e89e2236d94250c P d4eea540f3c241c1ba6290ca53769163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:08.021376 24825 ts_tablet_manager.cc:1367] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20240503 14:12:08.023196 24822 ts_tablet_manager.cc:1398] T d70414f531ac463c9e89e2236d94250c P d4eea540f3c241c1ba6290ca53769163: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240503 14:12:08.024003 24825 raft_consensus.cc:348] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:08.024235 24825 raft_consensus.cc:374] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.024554 24825 raft_consensus.cc:729] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.024924 24825 consensus_queue.cc:260] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:08.025794 24824 tablet_bootstrap.cc:492] T 916b8dad2a52497799b76127ce499a2f P c9a573f6faf64131b114af1e33c1b308: No bootstrap required, opened a new log
I20240503 14:12:08.026011 24824 ts_tablet_manager.cc:1367] T 916b8dad2a52497799b76127ce499a2f P c9a573f6faf64131b114af1e33c1b308: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20240503 14:12:08.027494 24824 raft_consensus.cc:348] T 916b8dad2a52497799b76127ce499a2f P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:08.027796 24824 raft_consensus.cc:374] T 916b8dad2a52497799b76127ce499a2f P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.027878 24824 raft_consensus.cc:729] T 916b8dad2a52497799b76127ce499a2f P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.028280 24824 consensus_queue.cc:260] T 916b8dad2a52497799b76127ce499a2f P c9a573f6faf64131b114af1e33c1b308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:08.029016 24824 ts_tablet_manager.cc:1398] T 916b8dad2a52497799b76127ce499a2f P c9a573f6faf64131b114af1e33c1b308: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:08.030691 24825 ts_tablet_manager.cc:1398] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:12:08.035933 24755 raft_consensus.cc:482] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:08.036108 24755 raft_consensus.cc:504] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:08.036299 24819 tablet_bootstrap.cc:492] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b: No bootstrap required, opened a new log
I20240503 14:12:08.036453 24819 ts_tablet_manager.cc:1367] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b: Time spent bootstrapping tablet: real 0.044s user 0.021s sys 0.020s
I20240503 14:12:08.036761 24823 tablet_bootstrap.cc:492] T d70414f531ac463c9e89e2236d94250c P 1e79cc5c647f476c98101f9e48b22b9e: No bootstrap required, opened a new log
I20240503 14:12:08.036924 24823 ts_tablet_manager.cc:1367] T d70414f531ac463c9e89e2236d94250c P 1e79cc5c647f476c98101f9e48b22b9e: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.012s
I20240503 14:12:08.037088 24755 leader_election.cc:288] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697), c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475)
I20240503 14:12:08.037235 24819 raft_consensus.cc:348] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:08.037456 24819 raft_consensus.cc:374] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.037600 24819 raft_consensus.cc:729] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.037781 24823 raft_consensus.cc:348] T d70414f531ac463c9e89e2236d94250c P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:08.038015 24823 raft_consensus.cc:374] T d70414f531ac463c9e89e2236d94250c P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.038386 24205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916b8dad2a52497799b76127ce499a2f" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308" is_pre_election: true
I20240503 14:12:08.038475 24819 consensus_queue.cc:260] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:08.038610 24205 raft_consensus.cc:2454] T 916b8dad2a52497799b76127ce499a2f P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 0.
I20240503 14:12:08.038161 24823 raft_consensus.cc:729] T d70414f531ac463c9e89e2236d94250c P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.039140 24819 ts_tablet_manager.cc:1398] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:08.039160 24823 consensus_queue.cc:260] T d70414f531ac463c9e89e2236d94250c P 1e79cc5c647f476c98101f9e48b22b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:08.039639 24819 tablet_bootstrap.cc:492] T 916b8dad2a52497799b76127ce499a2f P d40aeb06cfcc4c3a897b40290d12a15b: Bootstrap starting.
I20240503 14:12:08.040244 24823 ts_tablet_manager.cc:1398] T d70414f531ac463c9e89e2236d94250c P 1e79cc5c647f476c98101f9e48b22b9e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:08.040725 23615 leader_election.cc:302] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8795a3c2a7014baf9126dea0a6ce1bb4, c9a573f6faf64131b114af1e33c1b308; no voters:
I20240503 14:12:08.041051 24755 raft_consensus.cc:2781] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:08.041167 24755 raft_consensus.cc:482] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:08.041257 24755 raft_consensus.cc:3037] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.041424 24819 tablet_bootstrap.cc:654] T 916b8dad2a52497799b76127ce499a2f P d40aeb06cfcc4c3a897b40290d12a15b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.044106 24819 tablet_bootstrap.cc:492] T 916b8dad2a52497799b76127ce499a2f P d40aeb06cfcc4c3a897b40290d12a15b: No bootstrap required, opened a new log
I20240503 14:12:08.044323 24819 ts_tablet_manager.cc:1367] T 916b8dad2a52497799b76127ce499a2f P d40aeb06cfcc4c3a897b40290d12a15b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:12:08.045131 24819 raft_consensus.cc:348] T 916b8dad2a52497799b76127ce499a2f P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:08.045400 24819 raft_consensus.cc:374] T 916b8dad2a52497799b76127ce499a2f P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.045506 24819 raft_consensus.cc:729] T 916b8dad2a52497799b76127ce499a2f P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.045794 24819 consensus_queue.cc:260] T 916b8dad2a52497799b76127ce499a2f P d40aeb06cfcc4c3a897b40290d12a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:08.046553 24819 ts_tablet_manager.cc:1398] T 916b8dad2a52497799b76127ce499a2f P d40aeb06cfcc4c3a897b40290d12a15b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:08.046810 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916b8dad2a52497799b76127ce499a2f" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" is_pre_election: true
I20240503 14:12:08.046989 23944 raft_consensus.cc:2454] T 916b8dad2a52497799b76127ce499a2f P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 0.
I20240503 14:12:08.047076 24613 raft_consensus.cc:482] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:08.047232 24613 raft_consensus.cc:504] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:08.047873 24613 leader_election.cc:288] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675), d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473)
I20240503 14:12:08.048491 24075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d70414f531ac463c9e89e2236d94250c" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e" is_pre_election: true
I20240503 14:12:08.048507 23813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d70414f531ac463c9e89e2236d94250c" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163" is_pre_election: true
I20240503 14:12:08.048722 24075 raft_consensus.cc:2454] T d70414f531ac463c9e89e2236d94250c P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 0.
I20240503 14:12:08.048712 23813 raft_consensus.cc:2454] T d70414f531ac463c9e89e2236d94250c P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 0.
I20240503 14:12:08.049165 24273 leader_election.cc:302] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:08.049592 24613 raft_consensus.cc:2781] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:08.049719 24613 raft_consensus.cc:482] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:08.049822 24613 raft_consensus.cc:3037] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.067960 24613 raft_consensus.cc:504] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:08.067960 24755 raft_consensus.cc:504] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:08.069356 24205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916b8dad2a52497799b76127ce499a2f" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308"
I20240503 14:12:08.069449 23813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d70414f531ac463c9e89e2236d94250c" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163"
I20240503 14:12:08.069590 24205 raft_consensus.cc:3037] T 916b8dad2a52497799b76127ce499a2f P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.069584 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916b8dad2a52497799b76127ce499a2f" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b"
I20240503 14:12:08.069658 23813 raft_consensus.cc:3037] T d70414f531ac463c9e89e2236d94250c P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.069794 23944 raft_consensus.cc:3037] T 916b8dad2a52497799b76127ce499a2f P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.072542 23944 raft_consensus.cc:2454] T 916b8dad2a52497799b76127ce499a2f P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 1.
I20240503 14:12:08.072542 23813 raft_consensus.cc:2454] T d70414f531ac463c9e89e2236d94250c P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 1.
I20240503 14:12:08.072860 24613 leader_election.cc:288] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 election: Requested vote from peers 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675), d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473)
I20240503 14:12:08.072978 24075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d70414f531ac463c9e89e2236d94250c" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e"
I20240503 14:12:08.073038 24273 leader_election.cc:302] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:08.073204 24075 raft_consensus.cc:3037] T d70414f531ac463c9e89e2236d94250c P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.073350 24613 raft_consensus.cc:2781] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:08.073400 24755 leader_election.cc:288] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 election: Requested vote from peers d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697), c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475)
I20240503 14:12:08.073662 23614 leader_election.cc:302] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8795a3c2a7014baf9126dea0a6ce1bb4, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:08.073956 24613 raft_consensus.cc:686] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [term 1 LEADER]: Becoming Leader. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Running, Role: LEADER
I20240503 14:12:08.074062 24755 raft_consensus.cc:2781] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:08.074748 24755 raft_consensus.cc:686] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 LEADER]: Becoming Leader. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Running, Role: LEADER
I20240503 14:12:08.074841 24613 consensus_queue.cc:237] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:08.075356 24755 consensus_queue.cc:237] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:08.078058 23531 catalog_manager.cc:5484] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff53511345114c4ab97a9a8decef3a74 (127.22.242.70). New cstate: current_term: 1 leader_uuid: "ff53511345114c4ab97a9a8decef3a74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:08.078511 23540 catalog_manager.cc:5484] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:08.171996 24075 raft_consensus.cc:2454] T d70414f531ac463c9e89e2236d94250c P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 1.
I20240503 14:12:08.172003 24205 raft_consensus.cc:2454] T 916b8dad2a52497799b76127ce499a2f P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 1.
I20240503 14:12:08.286136 24493 raft_consensus.cc:482] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:08.286335 24493 raft_consensus.cc:504] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:08.287039 24493 leader_election.cc:288] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475), 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175)
I20240503 14:12:08.287477 24205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e304b5783546aea642a1525767007e" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308" is_pre_election: true
I20240503 14:12:08.287580 23680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e304b5783546aea642a1525767007e" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" is_pre_election: true
I20240503 14:12:08.287708 24205 raft_consensus.cc:2454] T 16e304b5783546aea642a1525767007e P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 0.
I20240503 14:12:08.287830 23680 raft_consensus.cc:2454] T 16e304b5783546aea642a1525767007e P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 0.
I20240503 14:12:08.288132 23878 leader_election.cc:302] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:08.288509 24493 raft_consensus.cc:2781] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:08.288619 24493 raft_consensus.cc:482] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:08.288705 24493 raft_consensus.cc:3037] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.290511 24493 raft_consensus.cc:504] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:08.291015 24493 leader_election.cc:288] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 election: Requested vote from peers c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475), 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175)
I20240503 14:12:08.291435 24205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e304b5783546aea642a1525767007e" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308"
I20240503 14:12:08.291471 23680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e304b5783546aea642a1525767007e" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4"
I20240503 14:12:08.291622 24205 raft_consensus.cc:3037] T 16e304b5783546aea642a1525767007e P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.291633 23680 raft_consensus.cc:3037] T 16e304b5783546aea642a1525767007e P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.293535 23680 raft_consensus.cc:2454] T 16e304b5783546aea642a1525767007e P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 1.
I20240503 14:12:08.293578 24205 raft_consensus.cc:2454] T 16e304b5783546aea642a1525767007e P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 1.
I20240503 14:12:08.293968 23878 leader_election.cc:302] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8795a3c2a7014baf9126dea0a6ce1bb4, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:08.294310 24493 raft_consensus.cc:2781] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:08.294698 24493 raft_consensus.cc:686] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [term 1 LEADER]: Becoming Leader. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Running, Role: LEADER
I20240503 14:12:08.294979 24493 consensus_queue.cc:237] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:08.297854 23540 catalog_manager.cc:5484] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b reported cstate change: term changed from 0 to 1, leader changed from <none> to d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67). New cstate: current_term: 1 leader_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:08.470633 24755 consensus_queue.cc:1035] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:08.477949 24755 consensus_queue.cc:1035] T 916b8dad2a52497799b76127ce499a2f P 8795a3c2a7014baf9126dea0a6ce1bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:08.538118 24613 consensus_queue.cc:1035] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:08.545806 24613 consensus_queue.cc:1035] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:08.698343 24842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:08.698719 24842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:08.706749 24842 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 14:12:08.782459 24493 consensus_queue.cc:1035] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:08.798270 24840 consensus_queue.cc:1035] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 253.325 ms
time per row: 0.126663 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:12:10.071168 23538 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33990:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:10.080240 23792 tablet_service.cc:1467] Processing CreateTablet for tablet 9e6476b88ba546cc82d721f84d6084df (DEFAULT_TABLE table=rebalance_test_table_5 [id=507b7379426a4468920e38a261f947cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.080372 24054 tablet_service.cc:1467] Processing CreateTablet for tablet 9e6476b88ba546cc82d721f84d6084df (DEFAULT_TABLE table=rebalance_test_table_5 [id=507b7379426a4468920e38a261f947cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.080749 23792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e6476b88ba546cc82d721f84d6084df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.080806 24054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e6476b88ba546cc82d721f84d6084df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.081418 24055 tablet_service.cc:1467] Processing CreateTablet for tablet fa04297e5841498caad1c4bf4b851fe4 (DEFAULT_TABLE table=rebalance_test_table_5 [id=507b7379426a4468920e38a261f947cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.081537 23793 tablet_service.cc:1467] Processing CreateTablet for tablet fa04297e5841498caad1c4bf4b851fe4 (DEFAULT_TABLE table=rebalance_test_table_5 [id=507b7379426a4468920e38a261f947cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.081833 24055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa04297e5841498caad1c4bf4b851fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.081949 23793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa04297e5841498caad1c4bf4b851fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.084817 24316 tablet_service.cc:1467] Processing CreateTablet for tablet 9e6476b88ba546cc82d721f84d6084df (DEFAULT_TABLE table=rebalance_test_table_5 [id=507b7379426a4468920e38a261f947cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.085225 24316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e6476b88ba546cc82d721f84d6084df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.088618 24317 tablet_service.cc:1467] Processing CreateTablet for tablet fa04297e5841498caad1c4bf4b851fe4 (DEFAULT_TABLE table=rebalance_test_table_5 [id=507b7379426a4468920e38a261f947cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.089044 24317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa04297e5841498caad1c4bf4b851fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.091003 24916 tablet_bootstrap.cc:492] T 9e6476b88ba546cc82d721f84d6084df P 1e79cc5c647f476c98101f9e48b22b9e: Bootstrap starting.
I20240503 14:12:10.092600 23924 tablet_service.cc:1467] Processing CreateTablet for tablet caf778976e774eabab30aa682d6bd5a0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=507b7379426a4468920e38a261f947cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.093034 23924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf778976e774eabab30aa682d6bd5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.095012 24916 tablet_bootstrap.cc:654] T 9e6476b88ba546cc82d721f84d6084df P 1e79cc5c647f476c98101f9e48b22b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.097154 23660 tablet_service.cc:1467] Processing CreateTablet for tablet caf778976e774eabab30aa682d6bd5a0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=507b7379426a4468920e38a261f947cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.097563 23660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf778976e774eabab30aa682d6bd5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.097810 24186 tablet_service.cc:1467] Processing CreateTablet for tablet caf778976e774eabab30aa682d6bd5a0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=507b7379426a4468920e38a261f947cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.098327 24186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf778976e774eabab30aa682d6bd5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.102098 24915 tablet_bootstrap.cc:492] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163: Bootstrap starting.
I20240503 14:12:10.103780 24915 tablet_bootstrap.cc:654] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.104112 24919 tablet_bootstrap.cc:492] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4: Bootstrap starting.
I20240503 14:12:10.105120 24918 tablet_bootstrap.cc:492] T 9e6476b88ba546cc82d721f84d6084df P ff53511345114c4ab97a9a8decef3a74: Bootstrap starting.
I20240503 14:12:10.105895 24919 tablet_bootstrap.cc:654] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.106794 24918 tablet_bootstrap.cc:654] T 9e6476b88ba546cc82d721f84d6084df P ff53511345114c4ab97a9a8decef3a74: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.109036 24921 tablet_bootstrap.cc:492] T caf778976e774eabab30aa682d6bd5a0 P c9a573f6faf64131b114af1e33c1b308: Bootstrap starting.
I20240503 14:12:10.110531 24924 tablet_bootstrap.cc:492] T caf778976e774eabab30aa682d6bd5a0 P d40aeb06cfcc4c3a897b40290d12a15b: Bootstrap starting.
I20240503 14:12:10.111177 24921 tablet_bootstrap.cc:654] T caf778976e774eabab30aa682d6bd5a0 P c9a573f6faf64131b114af1e33c1b308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.112587 24924 tablet_bootstrap.cc:654] T caf778976e774eabab30aa682d6bd5a0 P d40aeb06cfcc4c3a897b40290d12a15b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.112708 24919 tablet_bootstrap.cc:492] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4: No bootstrap required, opened a new log
I20240503 14:12:10.112962 24919 ts_tablet_manager.cc:1367] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20240503 14:12:10.114022 24919 raft_consensus.cc:348] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:10.114461 24919 raft_consensus.cc:374] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.114544 24918 tablet_bootstrap.cc:492] T 9e6476b88ba546cc82d721f84d6084df P ff53511345114c4ab97a9a8decef3a74: No bootstrap required, opened a new log
I20240503 14:12:10.114676 24919 raft_consensus.cc:729] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.114697 24918 ts_tablet_manager.cc:1367] T 9e6476b88ba546cc82d721f84d6084df P ff53511345114c4ab97a9a8decef3a74: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20240503 14:12:10.115079 24919 consensus_queue.cc:260] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:10.115765 24918 raft_consensus.cc:348] T 9e6476b88ba546cc82d721f84d6084df P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:10.115943 24919 ts_tablet_manager.cc:1398] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:10.116674 24918 raft_consensus.cc:374] T 9e6476b88ba546cc82d721f84d6084df P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.116881 24918 raft_consensus.cc:729] T 9e6476b88ba546cc82d721f84d6084df P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.117187 24918 consensus_queue.cc:260] T 9e6476b88ba546cc82d721f84d6084df P ff53511345114c4ab97a9a8decef3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:10.117807 24918 ts_tablet_manager.cc:1398] T 9e6476b88ba546cc82d721f84d6084df P ff53511345114c4ab97a9a8decef3a74: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:10.118419 24924 tablet_bootstrap.cc:492] T caf778976e774eabab30aa682d6bd5a0 P d40aeb06cfcc4c3a897b40290d12a15b: No bootstrap required, opened a new log
I20240503 14:12:10.118476 24915 tablet_bootstrap.cc:492] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163: No bootstrap required, opened a new log
I20240503 14:12:10.118557 24924 ts_tablet_manager.cc:1367] T caf778976e774eabab30aa682d6bd5a0 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240503 14:12:10.118604 24915 ts_tablet_manager.cc:1367] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 14:12:10.119318 24915 raft_consensus.cc:348] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:10.119308 24924 raft_consensus.cc:348] T caf778976e774eabab30aa682d6bd5a0 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:10.119504 24915 raft_consensus.cc:374] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.119504 24924 raft_consensus.cc:374] T caf778976e774eabab30aa682d6bd5a0 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.119589 24924 raft_consensus.cc:729] T caf778976e774eabab30aa682d6bd5a0 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.119589 24915 raft_consensus.cc:729] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.119817 24915 consensus_queue.cc:260] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:10.119874 24924 consensus_queue.cc:260] T caf778976e774eabab30aa682d6bd5a0 P d40aeb06cfcc4c3a897b40290d12a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:10.120364 24915 ts_tablet_manager.cc:1398] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:10.121678 24918 tablet_bootstrap.cc:492] T fa04297e5841498caad1c4bf4b851fe4 P ff53511345114c4ab97a9a8decef3a74: Bootstrap starting.
I20240503 14:12:10.123795 24918 tablet_bootstrap.cc:654] T fa04297e5841498caad1c4bf4b851fe4 P ff53511345114c4ab97a9a8decef3a74: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.123801 24916 tablet_bootstrap.cc:492] T 9e6476b88ba546cc82d721f84d6084df P 1e79cc5c647f476c98101f9e48b22b9e: No bootstrap required, opened a new log
I20240503 14:12:10.123963 24916 ts_tablet_manager.cc:1367] T 9e6476b88ba546cc82d721f84d6084df P 1e79cc5c647f476c98101f9e48b22b9e: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.000s
I20240503 14:12:10.124855 24916 raft_consensus.cc:348] T 9e6476b88ba546cc82d721f84d6084df P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:10.125075 24916 raft_consensus.cc:374] T 9e6476b88ba546cc82d721f84d6084df P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.125131 24915 tablet_bootstrap.cc:492] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163: Bootstrap starting.
I20240503 14:12:10.125178 24916 raft_consensus.cc:729] T 9e6476b88ba546cc82d721f84d6084df P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.125385 24924 ts_tablet_manager.cc:1398] T caf778976e774eabab30aa682d6bd5a0 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:12:10.125429 24916 consensus_queue.cc:260] T 9e6476b88ba546cc82d721f84d6084df P 1e79cc5c647f476c98101f9e48b22b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:10.125960 24916 ts_tablet_manager.cc:1398] T 9e6476b88ba546cc82d721f84d6084df P 1e79cc5c647f476c98101f9e48b22b9e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:10.126567 24916 tablet_bootstrap.cc:492] T fa04297e5841498caad1c4bf4b851fe4 P 1e79cc5c647f476c98101f9e48b22b9e: Bootstrap starting.
I20240503 14:12:10.126739 24915 tablet_bootstrap.cc:654] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.128700 24916 tablet_bootstrap.cc:654] T fa04297e5841498caad1c4bf4b851fe4 P 1e79cc5c647f476c98101f9e48b22b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.130638 24918 tablet_bootstrap.cc:492] T fa04297e5841498caad1c4bf4b851fe4 P ff53511345114c4ab97a9a8decef3a74: No bootstrap required, opened a new log
I20240503 14:12:10.130856 24918 ts_tablet_manager.cc:1367] T fa04297e5841498caad1c4bf4b851fe4 P ff53511345114c4ab97a9a8decef3a74: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20240503 14:12:10.131870 24918 raft_consensus.cc:348] T fa04297e5841498caad1c4bf4b851fe4 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:10.132117 24918 raft_consensus.cc:374] T fa04297e5841498caad1c4bf4b851fe4 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.132243 24918 raft_consensus.cc:729] T fa04297e5841498caad1c4bf4b851fe4 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.132547 24918 consensus_queue.cc:260] T fa04297e5841498caad1c4bf4b851fe4 P ff53511345114c4ab97a9a8decef3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:10.133157 24918 ts_tablet_manager.cc:1398] T fa04297e5841498caad1c4bf4b851fe4 P ff53511345114c4ab97a9a8decef3a74: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:10.134447 24921 tablet_bootstrap.cc:492] T caf778976e774eabab30aa682d6bd5a0 P c9a573f6faf64131b114af1e33c1b308: No bootstrap required, opened a new log
I20240503 14:12:10.134444 24915 tablet_bootstrap.cc:492] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163: No bootstrap required, opened a new log
I20240503 14:12:10.134537 24916 tablet_bootstrap.cc:492] T fa04297e5841498caad1c4bf4b851fe4 P 1e79cc5c647f476c98101f9e48b22b9e: No bootstrap required, opened a new log
I20240503 14:12:10.134613 24915 ts_tablet_manager.cc:1367] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20240503 14:12:10.134711 24916 ts_tablet_manager.cc:1367] T fa04297e5841498caad1c4bf4b851fe4 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240503 14:12:10.134742 24921 ts_tablet_manager.cc:1367] T caf778976e774eabab30aa682d6bd5a0 P c9a573f6faf64131b114af1e33c1b308: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.000s
I20240503 14:12:10.135440 24915 raft_consensus.cc:348] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:10.135530 24916 raft_consensus.cc:348] T fa04297e5841498caad1c4bf4b851fe4 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:10.135759 24916 raft_consensus.cc:374] T fa04297e5841498caad1c4bf4b851fe4 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.135547 24921 raft_consensus.cc:348] T caf778976e774eabab30aa682d6bd5a0 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:10.135852 24916 raft_consensus.cc:729] T fa04297e5841498caad1c4bf4b851fe4 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.135986 24921 raft_consensus.cc:374] T caf778976e774eabab30aa682d6bd5a0 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.136135 24921 raft_consensus.cc:729] T caf778976e774eabab30aa682d6bd5a0 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.136085 24916 consensus_queue.cc:260] T fa04297e5841498caad1c4bf4b851fe4 P 1e79cc5c647f476c98101f9e48b22b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:10.135673 24915 raft_consensus.cc:374] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.136512 24915 raft_consensus.cc:729] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.136402 24921 consensus_queue.cc:260] T caf778976e774eabab30aa682d6bd5a0 P c9a573f6faf64131b114af1e33c1b308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:10.136654 24916 ts_tablet_manager.cc:1398] T fa04297e5841498caad1c4bf4b851fe4 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:10.136909 24915 consensus_queue.cc:260] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:10.137056 24921 ts_tablet_manager.cc:1398] T caf778976e774eabab30aa682d6bd5a0 P c9a573f6faf64131b114af1e33c1b308: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:10.137461 24915 ts_tablet_manager.cc:1398] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:12:10.284418 24711 raft_consensus.cc:482] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.284629 24711 raft_consensus.cc:504] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:10.285331 24711 leader_election.cc:288] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675), ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067)
I20240503 14:12:10.285928 24075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e6476b88ba546cc82d721f84d6084df" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e" is_pre_election: true
I20240503 14:12:10.285939 24336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e6476b88ba546cc82d721f84d6084df" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74" is_pre_election: true
I20240503 14:12:10.286173 24336 raft_consensus.cc:2454] T 9e6476b88ba546cc82d721f84d6084df P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 0.
I20240503 14:12:10.286201 24075 raft_consensus.cc:2454] T 9e6476b88ba546cc82d721f84d6084df P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 0.
I20240503 14:12:10.286640 23749 leader_election.cc:302] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, d4eea540f3c241c1ba6290ca53769163; no voters:
I20240503 14:12:10.286890 24711 raft_consensus.cc:2781] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:10.287003 24711 raft_consensus.cc:482] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:10.287073 24711 raft_consensus.cc:3037] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.288975 24711 raft_consensus.cc:504] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:10.289609 24711 leader_election.cc:288] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 election: Requested vote from peers 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675), ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067)
I20240503 14:12:10.290021 24075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e6476b88ba546cc82d721f84d6084df" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e"
I20240503 14:12:10.290154 24336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e6476b88ba546cc82d721f84d6084df" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74"
I20240503 14:12:10.290266 24075 raft_consensus.cc:3037] T 9e6476b88ba546cc82d721f84d6084df P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.290371 24336 raft_consensus.cc:3037] T 9e6476b88ba546cc82d721f84d6084df P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.293169 24075 raft_consensus.cc:2454] T 9e6476b88ba546cc82d721f84d6084df P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 1.
I20240503 14:12:10.293183 24336 raft_consensus.cc:2454] T 9e6476b88ba546cc82d721f84d6084df P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 1.
I20240503 14:12:10.293628 23749 leader_election.cc:302] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, d4eea540f3c241c1ba6290ca53769163; no voters:
I20240503 14:12:10.293951 24711 raft_consensus.cc:2781] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:10.294317 24711 raft_consensus.cc:686] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [term 1 LEADER]: Becoming Leader. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Running, Role: LEADER
I20240503 14:12:10.294692 24711 consensus_queue.cc:237] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:10.297650 23537 catalog_manager.cc:5484] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4eea540f3c241c1ba6290ca53769163 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "d4eea540f3c241c1ba6290ca53769163" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:10.305294 24711 raft_consensus.cc:482] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.305509 24711 raft_consensus.cc:504] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:10.306196 24711 leader_election.cc:288] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067), 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675)
I20240503 14:12:10.306741 24336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa04297e5841498caad1c4bf4b851fe4" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74" is_pre_election: true
I20240503 14:12:10.306824 24075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa04297e5841498caad1c4bf4b851fe4" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e" is_pre_election: true
I20240503 14:12:10.307015 24336 raft_consensus.cc:2454] T fa04297e5841498caad1c4bf4b851fe4 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 0.
I20240503 14:12:10.307042 24075 raft_consensus.cc:2454] T fa04297e5841498caad1c4bf4b851fe4 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 0.
I20240503 14:12:10.307459 23749 leader_election.cc:302] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:10.307757 24711 raft_consensus.cc:2781] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:10.307849 24711 raft_consensus.cc:482] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:10.307957 24711 raft_consensus.cc:3037] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.309831 24711 raft_consensus.cc:504] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:10.310416 24711 leader_election.cc:288] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 election: Requested vote from peers ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067), 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675)
I20240503 14:12:10.310874 24075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa04297e5841498caad1c4bf4b851fe4" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e"
I20240503 14:12:10.311084 24075 raft_consensus.cc:3037] T fa04297e5841498caad1c4bf4b851fe4 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.311041 24336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa04297e5841498caad1c4bf4b851fe4" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74"
I20240503 14:12:10.311254 24336 raft_consensus.cc:3037] T fa04297e5841498caad1c4bf4b851fe4 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.313915 24336 raft_consensus.cc:2454] T fa04297e5841498caad1c4bf4b851fe4 P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 1.
I20240503 14:12:10.313930 24075 raft_consensus.cc:2454] T fa04297e5841498caad1c4bf4b851fe4 P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 1.
I20240503 14:12:10.314364 23749 leader_election.cc:302] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, d4eea540f3c241c1ba6290ca53769163; no voters:
I20240503 14:12:10.314641 24711 raft_consensus.cc:2781] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:10.314819 24711 raft_consensus.cc:686] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [term 1 LEADER]: Becoming Leader. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Running, Role: LEADER
I20240503 14:12:10.315116 24711 consensus_queue.cc:237] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:10.318068 23537 catalog_manager.cc:5484] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4eea540f3c241c1ba6290ca53769163 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "d4eea540f3c241c1ba6290ca53769163" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:10.344504 24755 raft_consensus.cc:482] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.344727 24755 raft_consensus.cc:504] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:10.345402 24755 leader_election.cc:288] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697), c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475)
I20240503 14:12:10.345844 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf778976e774eabab30aa682d6bd5a0" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" is_pre_election: true
I20240503 14:12:10.345875 24206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf778976e774eabab30aa682d6bd5a0" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308" is_pre_election: true
I20240503 14:12:10.346074 23944 raft_consensus.cc:2454] T caf778976e774eabab30aa682d6bd5a0 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 0.
I20240503 14:12:10.346076 24206 raft_consensus.cc:2454] T caf778976e774eabab30aa682d6bd5a0 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 0.
I20240503 14:12:10.346503 23614 leader_election.cc:302] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8795a3c2a7014baf9126dea0a6ce1bb4, c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:10.346781 24755 raft_consensus.cc:2781] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:10.346880 24755 raft_consensus.cc:482] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:10.346990 24755 raft_consensus.cc:3037] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.348937 24755 raft_consensus.cc:504] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:10.349536 24755 leader_election.cc:288] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 election: Requested vote from peers d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697), c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475)
I20240503 14:12:10.349890 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf778976e774eabab30aa682d6bd5a0" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b"
I20240503 14:12:10.350076 23944 raft_consensus.cc:3037] T caf778976e774eabab30aa682d6bd5a0 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.351900 23944 raft_consensus.cc:2454] T caf778976e774eabab30aa682d6bd5a0 P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 1.
I20240503 14:12:10.352245 24206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf778976e774eabab30aa682d6bd5a0" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308"
I20240503 14:12:10.352301 23614 leader_election.cc:302] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8795a3c2a7014baf9126dea0a6ce1bb4, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:10.352442 24206 raft_consensus.cc:3037] T caf778976e774eabab30aa682d6bd5a0 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.352571 24755 raft_consensus.cc:2781] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:10.352965 24755 raft_consensus.cc:686] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 LEADER]: Becoming Leader. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Running, Role: LEADER
I20240503 14:12:10.353541 24755 consensus_queue.cc:237] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:10.355046 24206 raft_consensus.cc:2454] T caf778976e774eabab30aa682d6bd5a0 P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 1.
I20240503 14:12:10.356456 23537 catalog_manager.cc:5484] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:10.731997 24936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:10.732352 24936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:10.740526 24936 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 14:12:10.835384 24711 consensus_queue.cc:1035] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:10.859804 24967 consensus_queue.cc:1035] T fa04297e5841498caad1c4bf4b851fe4 P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:10.913115 24755 consensus_queue.cc:1035] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:10.914299 24711 consensus_queue.cc:1035] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:10.915220 24711 consensus_queue.cc:1035] T 9e6476b88ba546cc82d721f84d6084df P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:10.930090 24755 consensus_queue.cc:1035] T caf778976e774eabab30aa682d6bd5a0 P 8795a3c2a7014baf9126dea0a6ce1bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:10.985055 24972 mvcc.cc:205] Tried to move back new op lower bound from 7023597694776786944 to 7023597692330545152. Current Snapshot: MvccSnapshot[applied={T|T < 7023597694776786944}]
I20240503 14:12:10.998032 24973 mvcc.cc:205] Tried to move back new op lower bound from 7023597694909521920 to 7023597692089372672. Current Snapshot: MvccSnapshot[applied={T|T < 7023597694909521920}]
INSERT report
rows total: 2000
time total: 273.757 ms
time per row: 0.136879 ms
I20240503 14:12:11.160387 23537 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33990:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:11.169713 24186 tablet_service.cc:1467] Processing CreateTablet for tablet e99fc624e14a4e17a010e42e6c3960a2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=87ab680d6f8343f3ad7c4f3527bd73b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.169957 23924 tablet_service.cc:1467] Processing CreateTablet for tablet e99fc624e14a4e17a010e42e6c3960a2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=87ab680d6f8343f3ad7c4f3527bd73b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.170178 24186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99fc624e14a4e17a010e42e6c3960a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.170383 23924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99fc624e14a4e17a010e42e6c3960a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.171816 23792 tablet_service.cc:1467] Processing CreateTablet for tablet f08228ffd3984f45afd068fab3eb9a43 (DEFAULT_TABLE table=rebalance_test_table_6 [id=87ab680d6f8343f3ad7c4f3527bd73b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.172248 23792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f08228ffd3984f45afd068fab3eb9a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.172631 23661 tablet_service.cc:1467] Processing CreateTablet for tablet 7159aeda34f945cd8726b2b7e9ccfdff (DEFAULT_TABLE table=rebalance_test_table_6 [id=87ab680d6f8343f3ad7c4f3527bd73b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.173009 23661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7159aeda34f945cd8726b2b7e9ccfdff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.173988 24185 tablet_service.cc:1467] Processing CreateTablet for tablet 7159aeda34f945cd8726b2b7e9ccfdff (DEFAULT_TABLE table=rebalance_test_table_6 [id=87ab680d6f8343f3ad7c4f3527bd73b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.174366 24185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7159aeda34f945cd8726b2b7e9ccfdff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.174896 23660 tablet_service.cc:1467] Processing CreateTablet for tablet e99fc624e14a4e17a010e42e6c3960a2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=87ab680d6f8343f3ad7c4f3527bd73b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.175246 23660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99fc624e14a4e17a010e42e6c3960a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.176594 24317 tablet_service.cc:1467] Processing CreateTablet for tablet f08228ffd3984f45afd068fab3eb9a43 (DEFAULT_TABLE table=rebalance_test_table_6 [id=87ab680d6f8343f3ad7c4f3527bd73b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.177012 24317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f08228ffd3984f45afd068fab3eb9a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.172147 24055 tablet_service.cc:1467] Processing CreateTablet for tablet f08228ffd3984f45afd068fab3eb9a43 (DEFAULT_TABLE table=rebalance_test_table_6 [id=87ab680d6f8343f3ad7c4f3527bd73b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.179625 24055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f08228ffd3984f45afd068fab3eb9a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.181229 23923 tablet_service.cc:1467] Processing CreateTablet for tablet 7159aeda34f945cd8726b2b7e9ccfdff (DEFAULT_TABLE table=rebalance_test_table_6 [id=87ab680d6f8343f3ad7c4f3527bd73b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.181716 23923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7159aeda34f945cd8726b2b7e9ccfdff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.187312 25006 tablet_bootstrap.cc:492] T 7159aeda34f945cd8726b2b7e9ccfdff P 8795a3c2a7014baf9126dea0a6ce1bb4: Bootstrap starting.
I20240503 14:12:11.190115 25006 tablet_bootstrap.cc:654] T 7159aeda34f945cd8726b2b7e9ccfdff P 8795a3c2a7014baf9126dea0a6ce1bb4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:11.190446 25008 tablet_bootstrap.cc:492] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e: Bootstrap starting.
I20240503 14:12:11.190882 25005 tablet_bootstrap.cc:492] T f08228ffd3984f45afd068fab3eb9a43 P d4eea540f3c241c1ba6290ca53769163: Bootstrap starting.
I20240503 14:12:11.193200 25005 tablet_bootstrap.cc:654] T f08228ffd3984f45afd068fab3eb9a43 P d4eea540f3c241c1ba6290ca53769163: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:11.193210 25008 tablet_bootstrap.cc:654] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:11.203553 25008 tablet_bootstrap.cc:492] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e: No bootstrap required, opened a new log
I20240503 14:12:11.203807 25008 ts_tablet_manager.cc:1367] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240503 14:12:11.203862 25005 tablet_bootstrap.cc:492] T f08228ffd3984f45afd068fab3eb9a43 P d4eea540f3c241c1ba6290ca53769163: No bootstrap required, opened a new log
I20240503 14:12:11.204156 25005 ts_tablet_manager.cc:1367] T f08228ffd3984f45afd068fab3eb9a43 P d4eea540f3c241c1ba6290ca53769163: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240503 14:12:11.204636 25008 raft_consensus.cc:348] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:11.204931 25008 raft_consensus.cc:374] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.205067 25008 raft_consensus.cc:729] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.205214 25005 raft_consensus.cc:348] T f08228ffd3984f45afd068fab3eb9a43 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:11.205360 25008 consensus_queue.cc:260] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:11.205555 25005 raft_consensus.cc:374] T f08228ffd3984f45afd068fab3eb9a43 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.205711 25005 raft_consensus.cc:729] T f08228ffd3984f45afd068fab3eb9a43 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.206614 25008 ts_tablet_manager.cc:1398] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:12:11.206064 25005 consensus_queue.cc:260] T f08228ffd3984f45afd068fab3eb9a43 P d4eea540f3c241c1ba6290ca53769163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:11.209175 25005 ts_tablet_manager.cc:1398] T f08228ffd3984f45afd068fab3eb9a43 P d4eea540f3c241c1ba6290ca53769163: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:12:11.217372 25013 tablet_bootstrap.cc:492] T e99fc624e14a4e17a010e42e6c3960a2 P d40aeb06cfcc4c3a897b40290d12a15b: Bootstrap starting.
I20240503 14:12:11.217397 25012 tablet_bootstrap.cc:492] T f08228ffd3984f45afd068fab3eb9a43 P ff53511345114c4ab97a9a8decef3a74: Bootstrap starting.
I20240503 14:12:11.219125 25013 tablet_bootstrap.cc:654] T e99fc624e14a4e17a010e42e6c3960a2 P d40aeb06cfcc4c3a897b40290d12a15b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:11.221876 25007 tablet_bootstrap.cc:492] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308: Bootstrap starting.
I20240503 14:12:11.224062 25012 tablet_bootstrap.cc:654] T f08228ffd3984f45afd068fab3eb9a43 P ff53511345114c4ab97a9a8decef3a74: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:11.226358 25007 tablet_bootstrap.cc:654] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:11.226606 25006 tablet_bootstrap.cc:492] T 7159aeda34f945cd8726b2b7e9ccfdff P 8795a3c2a7014baf9126dea0a6ce1bb4: No bootstrap required, opened a new log
I20240503 14:12:11.226850 25006 ts_tablet_manager.cc:1367] T 7159aeda34f945cd8726b2b7e9ccfdff P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.018s
I20240503 14:12:11.227890 25006 raft_consensus.cc:348] T 7159aeda34f945cd8726b2b7e9ccfdff P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:11.228322 25006 raft_consensus.cc:374] T 7159aeda34f945cd8726b2b7e9ccfdff P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.228435 25012 tablet_bootstrap.cc:492] T f08228ffd3984f45afd068fab3eb9a43 P ff53511345114c4ab97a9a8decef3a74: No bootstrap required, opened a new log
I20240503 14:12:11.228494 25006 raft_consensus.cc:729] T 7159aeda34f945cd8726b2b7e9ccfdff P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.228629 25012 ts_tablet_manager.cc:1367] T f08228ffd3984f45afd068fab3eb9a43 P ff53511345114c4ab97a9a8decef3a74: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20240503 14:12:11.228885 25006 consensus_queue.cc:260] T 7159aeda34f945cd8726b2b7e9ccfdff P 8795a3c2a7014baf9126dea0a6ce1bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:11.229400 25012 raft_consensus.cc:348] T f08228ffd3984f45afd068fab3eb9a43 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:11.229669 25012 raft_consensus.cc:374] T f08228ffd3984f45afd068fab3eb9a43 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.229797 25012 raft_consensus.cc:729] T f08228ffd3984f45afd068fab3eb9a43 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.230065 25012 consensus_queue.cc:260] T f08228ffd3984f45afd068fab3eb9a43 P ff53511345114c4ab97a9a8decef3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:11.230639 25006 ts_tablet_manager.cc:1398] T 7159aeda34f945cd8726b2b7e9ccfdff P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:12:11.231086 25006 tablet_bootstrap.cc:492] T e99fc624e14a4e17a010e42e6c3960a2 P 8795a3c2a7014baf9126dea0a6ce1bb4: Bootstrap starting.
I20240503 14:12:11.233240 25007 tablet_bootstrap.cc:492] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308: No bootstrap required, opened a new log
I20240503 14:12:11.233466 25007 ts_tablet_manager.cc:1367] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240503 14:12:11.233510 25012 ts_tablet_manager.cc:1398] T f08228ffd3984f45afd068fab3eb9a43 P ff53511345114c4ab97a9a8decef3a74: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 14:12:11.233798 25006 tablet_bootstrap.cc:654] T e99fc624e14a4e17a010e42e6c3960a2 P 8795a3c2a7014baf9126dea0a6ce1bb4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:11.235569 25007 raft_consensus.cc:348] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:11.235883 25007 raft_consensus.cc:374] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.236032 25007 raft_consensus.cc:729] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.236408 25007 consensus_queue.cc:260] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:11.238343 25007 ts_tablet_manager.cc:1398] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:11.238821 25007 tablet_bootstrap.cc:492] T 7159aeda34f945cd8726b2b7e9ccfdff P c9a573f6faf64131b114af1e33c1b308: Bootstrap starting.
I20240503 14:12:11.241860 25007 tablet_bootstrap.cc:654] T 7159aeda34f945cd8726b2b7e9ccfdff P c9a573f6faf64131b114af1e33c1b308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:11.243209 25013 tablet_bootstrap.cc:492] T e99fc624e14a4e17a010e42e6c3960a2 P d40aeb06cfcc4c3a897b40290d12a15b: No bootstrap required, opened a new log
I20240503 14:12:11.243415 25013 ts_tablet_manager.cc:1367] T e99fc624e14a4e17a010e42e6c3960a2 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.000s
I20240503 14:12:11.244423 25013 raft_consensus.cc:348] T e99fc624e14a4e17a010e42e6c3960a2 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:11.244678 25013 raft_consensus.cc:374] T e99fc624e14a4e17a010e42e6c3960a2 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.244891 25013 raft_consensus.cc:729] T e99fc624e14a4e17a010e42e6c3960a2 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.245170 25013 consensus_queue.cc:260] T e99fc624e14a4e17a010e42e6c3960a2 P d40aeb06cfcc4c3a897b40290d12a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:11.245764 25013 ts_tablet_manager.cc:1398] T e99fc624e14a4e17a010e42e6c3960a2 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:11.246573 25013 tablet_bootstrap.cc:492] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b: Bootstrap starting.
I20240503 14:12:11.249002 25013 tablet_bootstrap.cc:654] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:11.253453 25007 tablet_bootstrap.cc:492] T 7159aeda34f945cd8726b2b7e9ccfdff P c9a573f6faf64131b114af1e33c1b308: No bootstrap required, opened a new log
I20240503 14:12:11.253654 25007 ts_tablet_manager.cc:1367] T 7159aeda34f945cd8726b2b7e9ccfdff P c9a573f6faf64131b114af1e33c1b308: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20240503 14:12:11.254125 25013 tablet_bootstrap.cc:492] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b: No bootstrap required, opened a new log
I20240503 14:12:11.254132 25006 tablet_bootstrap.cc:492] T e99fc624e14a4e17a010e42e6c3960a2 P 8795a3c2a7014baf9126dea0a6ce1bb4: No bootstrap required, opened a new log
I20240503 14:12:11.254292 25013 ts_tablet_manager.cc:1367] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 14:12:11.254294 25006 ts_tablet_manager.cc:1367] T e99fc624e14a4e17a010e42e6c3960a2 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.002s
I20240503 14:12:11.254400 25007 raft_consensus.cc:348] T 7159aeda34f945cd8726b2b7e9ccfdff P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:11.254601 25007 raft_consensus.cc:374] T 7159aeda34f945cd8726b2b7e9ccfdff P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.254662 25007 raft_consensus.cc:729] T 7159aeda34f945cd8726b2b7e9ccfdff P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.254900 25007 consensus_queue.cc:260] T 7159aeda34f945cd8726b2b7e9ccfdff P c9a573f6faf64131b114af1e33c1b308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:11.255045 25006 raft_consensus.cc:348] T e99fc624e14a4e17a010e42e6c3960a2 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:11.255316 25006 raft_consensus.cc:374] T e99fc624e14a4e17a010e42e6c3960a2 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.255429 25006 raft_consensus.cc:729] T e99fc624e14a4e17a010e42e6c3960a2 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.255494 25013 raft_consensus.cc:348] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:11.255784 25013 raft_consensus.cc:374] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.255750 25006 consensus_queue.cc:260] T e99fc624e14a4e17a010e42e6c3960a2 P 8795a3c2a7014baf9126dea0a6ce1bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:11.255900 25013 raft_consensus.cc:729] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.256343 25006 ts_tablet_manager.cc:1398] T e99fc624e14a4e17a010e42e6c3960a2 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:11.256748 25007 ts_tablet_manager.cc:1398] T 7159aeda34f945cd8726b2b7e9ccfdff P c9a573f6faf64131b114af1e33c1b308: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:11.257104 25013 consensus_queue.cc:260] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:11.257624 25013 ts_tablet_manager.cc:1398] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:11.391984 24927 raft_consensus.cc:482] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:11.392201 24927 raft_consensus.cc:504] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:11.392906 24927 leader_election.cc:288] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475), 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175)
I20240503 14:12:11.393435 24206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7159aeda34f945cd8726b2b7e9ccfdff" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308" is_pre_election: true
I20240503 14:12:11.393479 23680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7159aeda34f945cd8726b2b7e9ccfdff" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" is_pre_election: true
I20240503 14:12:11.393630 24206 raft_consensus.cc:2454] T 7159aeda34f945cd8726b2b7e9ccfdff P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 0.
I20240503 14:12:11.393806 23680 raft_consensus.cc:2454] T 7159aeda34f945cd8726b2b7e9ccfdff P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 0.
I20240503 14:12:11.394115 23878 leader_election.cc:302] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:11.394397 24927 raft_consensus.cc:2781] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:11.394507 24927 raft_consensus.cc:482] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:11.394594 24927 raft_consensus.cc:3037] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.396559 24927 raft_consensus.cc:504] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:11.397084 24927 leader_election.cc:288] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 election: Requested vote from peers c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475), 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175)
I20240503 14:12:11.397553 24206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7159aeda34f945cd8726b2b7e9ccfdff" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308"
I20240503 14:12:11.397594 23680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7159aeda34f945cd8726b2b7e9ccfdff" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4"
I20240503 14:12:11.397732 24206 raft_consensus.cc:3037] T 7159aeda34f945cd8726b2b7e9ccfdff P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.397794 23680 raft_consensus.cc:3037] T 7159aeda34f945cd8726b2b7e9ccfdff P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.399545 23680 raft_consensus.cc:2454] T 7159aeda34f945cd8726b2b7e9ccfdff P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 1.
I20240503 14:12:11.399561 24206 raft_consensus.cc:2454] T 7159aeda34f945cd8726b2b7e9ccfdff P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 1.
I20240503 14:12:11.399940 23878 leader_election.cc:302] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:11.400250 24927 raft_consensus.cc:2781] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:11.400626 24927 raft_consensus.cc:686] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [term 1 LEADER]: Becoming Leader. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Running, Role: LEADER
I20240503 14:12:11.400908 24927 consensus_queue.cc:237] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:11.403849 23537 catalog_manager.cc:5484] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b reported cstate change: term changed from 0 to 1, leader changed from <none> to d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67). New cstate: current_term: 1 leader_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:11.419577 24757 raft_consensus.cc:482] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:11.419795 24757 raft_consensus.cc:504] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:11.420571 24757 leader_election.cc:288] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697), 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175)
I20240503 14:12:11.420940 23943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99fc624e14a4e17a010e42e6c3960a2" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" is_pre_election: true
I20240503 14:12:11.421142 23943 raft_consensus.cc:2454] T e99fc624e14a4e17a010e42e6c3960a2 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 0.
I20240503 14:12:11.421067 23680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99fc624e14a4e17a010e42e6c3960a2" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" is_pre_election: true
I20240503 14:12:11.421303 23680 raft_consensus.cc:2454] T e99fc624e14a4e17a010e42e6c3960a2 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 0.
I20240503 14:12:11.421619 24139 leader_election.cc:302] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:11.421938 24757 raft_consensus.cc:2781] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:11.422029 24757 raft_consensus.cc:482] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:11.422147 24757 raft_consensus.cc:3037] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.424019 24757 raft_consensus.cc:504] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:11.424607 24757 leader_election.cc:288] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 election: Requested vote from peers d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697), 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175)
I20240503 14:12:11.425061 23943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99fc624e14a4e17a010e42e6c3960a2" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b"
I20240503 14:12:11.425182 23680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99fc624e14a4e17a010e42e6c3960a2" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4"
I20240503 14:12:11.425269 23943 raft_consensus.cc:3037] T e99fc624e14a4e17a010e42e6c3960a2 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.425391 23680 raft_consensus.cc:3037] T e99fc624e14a4e17a010e42e6c3960a2 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.427563 23943 raft_consensus.cc:2454] T e99fc624e14a4e17a010e42e6c3960a2 P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 1.
I20240503 14:12:11.427884 23680 raft_consensus.cc:2454] T e99fc624e14a4e17a010e42e6c3960a2 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 1.
I20240503 14:12:11.427982 24139 leader_election.cc:302] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:11.428328 24757 raft_consensus.cc:2781] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:11.428776 24757 raft_consensus.cc:686] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [term 1 LEADER]: Becoming Leader. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Running, Role: LEADER
I20240503 14:12:11.429086 24757 consensus_queue.cc:237] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:11.431926 23537 catalog_manager.cc:5484] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9a573f6faf64131b114af1e33c1b308 (127.22.242.69). New cstate: current_term: 1 leader_uuid: "c9a573f6faf64131b114af1e33c1b308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:11.483002 24910 raft_consensus.cc:482] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:11.483155 24910 raft_consensus.cc:504] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:11.483848 24910 leader_election.cc:288] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473), ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067)
I20240503 14:12:11.484338 23813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08228ffd3984f45afd068fab3eb9a43" candidate_uuid: "1e79cc5c647f476c98101f9e48b22b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163" is_pre_election: true
I20240503 14:12:11.484359 24336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08228ffd3984f45afd068fab3eb9a43" candidate_uuid: "1e79cc5c647f476c98101f9e48b22b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74" is_pre_election: true
I20240503 14:12:11.484551 23813 raft_consensus.cc:2454] T f08228ffd3984f45afd068fab3eb9a43 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e79cc5c647f476c98101f9e48b22b9e in term 0.
I20240503 14:12:11.484555 24336 raft_consensus.cc:2454] T f08228ffd3984f45afd068fab3eb9a43 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e79cc5c647f476c98101f9e48b22b9e in term 0.
I20240503 14:12:11.484939 24011 leader_election.cc:302] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, d4eea540f3c241c1ba6290ca53769163; no voters:
I20240503 14:12:11.485220 24910 raft_consensus.cc:2781] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:11.485344 24910 raft_consensus.cc:482] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:11.485430 24910 raft_consensus.cc:3037] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.487334 24910 raft_consensus.cc:504] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:11.487824 24910 leader_election.cc:288] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [CANDIDATE]: Term 1 election: Requested vote from peers d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473), ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067)
I20240503 14:12:11.488250 23813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08228ffd3984f45afd068fab3eb9a43" candidate_uuid: "1e79cc5c647f476c98101f9e48b22b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163"
I20240503 14:12:11.488348 24336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08228ffd3984f45afd068fab3eb9a43" candidate_uuid: "1e79cc5c647f476c98101f9e48b22b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74"
I20240503 14:12:11.488440 23813 raft_consensus.cc:3037] T f08228ffd3984f45afd068fab3eb9a43 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.488543 24336 raft_consensus.cc:3037] T f08228ffd3984f45afd068fab3eb9a43 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.490317 24336 raft_consensus.cc:2454] T f08228ffd3984f45afd068fab3eb9a43 P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e79cc5c647f476c98101f9e48b22b9e in term 1.
I20240503 14:12:11.490442 23813 raft_consensus.cc:2454] T f08228ffd3984f45afd068fab3eb9a43 P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e79cc5c647f476c98101f9e48b22b9e in term 1.
I20240503 14:12:11.490725 24009 leader_election.cc:302] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:11.491007 24910 raft_consensus.cc:2781] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:11.491415 24910 raft_consensus.cc:686] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [term 1 LEADER]: Becoming Leader. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Running, Role: LEADER
I20240503 14:12:11.491724 24910 consensus_queue.cc:237] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:11.494511 23537 catalog_manager.cc:5484] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68). New cstate: current_term: 1 leader_uuid: "1e79cc5c647f476c98101f9e48b22b9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:11.837616 24757 consensus_queue.cc:1035] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:11.849623 25022 consensus_queue.cc:1035] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:11.883654 24927 consensus_queue.cc:1035] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:11.889822 25020 consensus_queue.cc:1035] T 7159aeda34f945cd8726b2b7e9ccfdff P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:11.895648 25026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:11.896220 25026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:11.906530 25026 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 14:12:12.075229 24910 consensus_queue.cc:1035] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:12.086643 24910 consensus_queue.cc:1035] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 327.514 ms
time per row: 0.163757 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:12:13.348294 23536 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33990:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:13.356823 23792 tablet_service.cc:1467] Processing CreateTablet for tablet 2cfc5cdabc784d1ea5f74b066f4e2be8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=02e1c7c491b84731bc5a054fc23ac1e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.357098 24055 tablet_service.cc:1467] Processing CreateTablet for tablet 2cfc5cdabc784d1ea5f74b066f4e2be8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=02e1c7c491b84731bc5a054fc23ac1e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.357285 23792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cfc5cdabc784d1ea5f74b066f4e2be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.357527 24055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cfc5cdabc784d1ea5f74b066f4e2be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.357439 24317 tablet_service.cc:1467] Processing CreateTablet for tablet 2cfc5cdabc784d1ea5f74b066f4e2be8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=02e1c7c491b84731bc5a054fc23ac1e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.357821 24317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cfc5cdabc784d1ea5f74b066f4e2be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.358256 23661 tablet_service.cc:1467] Processing CreateTablet for tablet def4596ad4634265a15ce77a54833853 (DEFAULT_TABLE table=rebalance_test_table_7 [id=02e1c7c491b84731bc5a054fc23ac1e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.358666 23661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet def4596ad4634265a15ce77a54833853. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.363961 24316 tablet_service.cc:1467] Processing CreateTablet for tablet e9c2d15fb0434a22afc3191ed8df3aac (DEFAULT_TABLE table=rebalance_test_table_7 [id=02e1c7c491b84731bc5a054fc23ac1e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.365274 24054 tablet_service.cc:1467] Processing CreateTablet for tablet e9c2d15fb0434a22afc3191ed8df3aac (DEFAULT_TABLE table=rebalance_test_table_7 [id=02e1c7c491b84731bc5a054fc23ac1e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.365499 23793 tablet_service.cc:1467] Processing CreateTablet for tablet e9c2d15fb0434a22afc3191ed8df3aac (DEFAULT_TABLE table=rebalance_test_table_7 [id=02e1c7c491b84731bc5a054fc23ac1e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.365628 24054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9c2d15fb0434a22afc3191ed8df3aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.365628 24316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9c2d15fb0434a22afc3191ed8df3aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.365847 23793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9c2d15fb0434a22afc3191ed8df3aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.368963 23924 tablet_service.cc:1467] Processing CreateTablet for tablet def4596ad4634265a15ce77a54833853 (DEFAULT_TABLE table=rebalance_test_table_7 [id=02e1c7c491b84731bc5a054fc23ac1e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.369377 23924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet def4596ad4634265a15ce77a54833853. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.370338 25103 tablet_bootstrap.cc:492] T def4596ad4634265a15ce77a54833853 P 8795a3c2a7014baf9126dea0a6ce1bb4: Bootstrap starting.
I20240503 14:12:13.371902 25103 tablet_bootstrap.cc:654] T def4596ad4634265a15ce77a54833853 P 8795a3c2a7014baf9126dea0a6ce1bb4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.372237 24186 tablet_service.cc:1467] Processing CreateTablet for tablet def4596ad4634265a15ce77a54833853 (DEFAULT_TABLE table=rebalance_test_table_7 [id=02e1c7c491b84731bc5a054fc23ac1e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.372635 24186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet def4596ad4634265a15ce77a54833853. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.377996 25100 tablet_bootstrap.cc:492] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P 1e79cc5c647f476c98101f9e48b22b9e: Bootstrap starting.
I20240503 14:12:13.378569 25102 tablet_bootstrap.cc:492] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P d4eea540f3c241c1ba6290ca53769163: Bootstrap starting.
I20240503 14:12:13.380204 25102 tablet_bootstrap.cc:654] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P d4eea540f3c241c1ba6290ca53769163: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.380652 25103 tablet_bootstrap.cc:492] T def4596ad4634265a15ce77a54833853 P 8795a3c2a7014baf9126dea0a6ce1bb4: No bootstrap required, opened a new log
I20240503 14:12:13.380805 25103 ts_tablet_manager.cc:1367] T def4596ad4634265a15ce77a54833853 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20240503 14:12:13.381775 25103 raft_consensus.cc:348] T def4596ad4634265a15ce77a54833853 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:13.382055 25103 raft_consensus.cc:374] T def4596ad4634265a15ce77a54833853 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.382155 25103 raft_consensus.cc:729] T def4596ad4634265a15ce77a54833853 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.382443 25103 consensus_queue.cc:260] T def4596ad4634265a15ce77a54833853 P 8795a3c2a7014baf9126dea0a6ce1bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:13.383949 25103 ts_tablet_manager.cc:1398] T def4596ad4634265a15ce77a54833853 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:12:13.384582 25100 tablet_bootstrap.cc:654] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P 1e79cc5c647f476c98101f9e48b22b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.384709 25106 tablet_bootstrap.cc:492] T def4596ad4634265a15ce77a54833853 P c9a573f6faf64131b114af1e33c1b308: Bootstrap starting.
I20240503 14:12:13.391590 25101 tablet_bootstrap.cc:492] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74: Bootstrap starting.
I20240503 14:12:13.393481 25106 tablet_bootstrap.cc:654] T def4596ad4634265a15ce77a54833853 P c9a573f6faf64131b114af1e33c1b308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.393589 25101 tablet_bootstrap.cc:654] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.395145 25105 tablet_bootstrap.cc:492] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b: Bootstrap starting.
I20240503 14:12:13.397409 25105 tablet_bootstrap.cc:654] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.397655 25102 tablet_bootstrap.cc:492] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P d4eea540f3c241c1ba6290ca53769163: No bootstrap required, opened a new log
I20240503 14:12:13.397938 25102 ts_tablet_manager.cc:1367] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P d4eea540f3c241c1ba6290ca53769163: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.010s
I20240503 14:12:13.398739 25102 raft_consensus.cc:348] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:13.398989 25100 tablet_bootstrap.cc:492] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P 1e79cc5c647f476c98101f9e48b22b9e: No bootstrap required, opened a new log
I20240503 14:12:13.399026 25102 raft_consensus.cc:374] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.399164 25102 raft_consensus.cc:729] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.399206 25100 ts_tablet_manager.cc:1367] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.016s
I20240503 14:12:13.399494 25102 consensus_queue.cc:260] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P d4eea540f3c241c1ba6290ca53769163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:13.400074 25100 raft_consensus.cc:348] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:13.400384 25100 raft_consensus.cc:374] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.400521 25100 raft_consensus.cc:729] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.401031 25102 ts_tablet_manager.cc:1398] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P d4eea540f3c241c1ba6290ca53769163: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:13.401031 25100 consensus_queue.cc:260] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P 1e79cc5c647f476c98101f9e48b22b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:13.401304 25102 tablet_bootstrap.cc:492] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163: Bootstrap starting.
I20240503 14:12:13.402561 25100 ts_tablet_manager.cc:1398] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:13.402920 25100 tablet_bootstrap.cc:492] T e9c2d15fb0434a22afc3191ed8df3aac P 1e79cc5c647f476c98101f9e48b22b9e: Bootstrap starting.
I20240503 14:12:13.403111 25102 tablet_bootstrap.cc:654] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.404713 25100 tablet_bootstrap.cc:654] T e9c2d15fb0434a22afc3191ed8df3aac P 1e79cc5c647f476c98101f9e48b22b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.405879 25105 tablet_bootstrap.cc:492] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b: No bootstrap required, opened a new log
I20240503 14:12:13.406085 25105 ts_tablet_manager.cc:1367] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240503 14:12:13.406988 25105 raft_consensus.cc:348] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:13.407292 25105 raft_consensus.cc:374] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.407439 25105 raft_consensus.cc:729] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.407735 25106 tablet_bootstrap.cc:492] T def4596ad4634265a15ce77a54833853 P c9a573f6faf64131b114af1e33c1b308: No bootstrap required, opened a new log
I20240503 14:12:13.407763 25105 consensus_queue.cc:260] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:13.407941 25106 ts_tablet_manager.cc:1367] T def4596ad4634265a15ce77a54833853 P c9a573f6faf64131b114af1e33c1b308: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20240503 14:12:13.408267 25101 tablet_bootstrap.cc:492] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74: No bootstrap required, opened a new log
I20240503 14:12:13.408466 25101 ts_tablet_manager.cc:1367] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 14:12:13.409495 25101 raft_consensus.cc:348] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:13.409760 25101 raft_consensus.cc:374] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.409837 25101 raft_consensus.cc:729] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.410248 25101 consensus_queue.cc:260] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:13.411044 25105 ts_tablet_manager.cc:1398] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:12:13.411770 25106 raft_consensus.cc:348] T def4596ad4634265a15ce77a54833853 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:13.412024 25106 raft_consensus.cc:374] T def4596ad4634265a15ce77a54833853 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.412199 25106 raft_consensus.cc:729] T def4596ad4634265a15ce77a54833853 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.412505 25106 consensus_queue.cc:260] T def4596ad4634265a15ce77a54833853 P c9a573f6faf64131b114af1e33c1b308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:13.413861 25101 ts_tablet_manager.cc:1398] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:13.414297 25101 tablet_bootstrap.cc:492] T e9c2d15fb0434a22afc3191ed8df3aac P ff53511345114c4ab97a9a8decef3a74: Bootstrap starting.
I20240503 14:12:13.415674 25106 ts_tablet_manager.cc:1398] T def4596ad4634265a15ce77a54833853 P c9a573f6faf64131b114af1e33c1b308: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:12:13.415777 25100 tablet_bootstrap.cc:492] T e9c2d15fb0434a22afc3191ed8df3aac P 1e79cc5c647f476c98101f9e48b22b9e: No bootstrap required, opened a new log
I20240503 14:12:13.415958 25100 ts_tablet_manager.cc:1367] T e9c2d15fb0434a22afc3191ed8df3aac P 1e79cc5c647f476c98101f9e48b22b9e: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.002s
I20240503 14:12:13.416482 25102 tablet_bootstrap.cc:492] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163: No bootstrap required, opened a new log
I20240503 14:12:13.416829 25102 ts_tablet_manager.cc:1367] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20240503 14:12:13.416900 25100 raft_consensus.cc:348] T e9c2d15fb0434a22afc3191ed8df3aac P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:13.417167 25100 raft_consensus.cc:374] T e9c2d15fb0434a22afc3191ed8df3aac P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.417256 25100 raft_consensus.cc:729] T e9c2d15fb0434a22afc3191ed8df3aac P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.417517 25100 consensus_queue.cc:260] T e9c2d15fb0434a22afc3191ed8df3aac P 1e79cc5c647f476c98101f9e48b22b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:13.417652 25101 tablet_bootstrap.cc:654] T e9c2d15fb0434a22afc3191ed8df3aac P ff53511345114c4ab97a9a8decef3a74: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.417635 25102 raft_consensus.cc:348] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:13.417867 25102 raft_consensus.cc:374] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.417955 25102 raft_consensus.cc:729] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.418255 25102 consensus_queue.cc:260] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:13.418857 25100 ts_tablet_manager.cc:1398] T e9c2d15fb0434a22afc3191ed8df3aac P 1e79cc5c647f476c98101f9e48b22b9e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:13.420706 25102 ts_tablet_manager.cc:1398] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:13.421392 25101 tablet_bootstrap.cc:492] T e9c2d15fb0434a22afc3191ed8df3aac P ff53511345114c4ab97a9a8decef3a74: No bootstrap required, opened a new log
I20240503 14:12:13.422533 25101 ts_tablet_manager.cc:1367] T e9c2d15fb0434a22afc3191ed8df3aac P ff53511345114c4ab97a9a8decef3a74: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240503 14:12:13.423286 25101 raft_consensus.cc:348] T e9c2d15fb0434a22afc3191ed8df3aac P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:13.423506 25101 raft_consensus.cc:374] T e9c2d15fb0434a22afc3191ed8df3aac P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.423594 25101 raft_consensus.cc:729] T e9c2d15fb0434a22afc3191ed8df3aac P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.423862 25101 consensus_queue.cc:260] T e9c2d15fb0434a22afc3191ed8df3aac P ff53511345114c4ab97a9a8decef3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:13.424386 25101 ts_tablet_manager.cc:1398] T e9c2d15fb0434a22afc3191ed8df3aac P ff53511345114c4ab97a9a8decef3a74: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:13.499588 24937 raft_consensus.cc:482] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:13.499766 24937 raft_consensus.cc:504] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:13.500425 24937 leader_election.cc:288] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473), 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675)
I20240503 14:12:13.500918 23812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cfc5cdabc784d1ea5f74b066f4e2be8" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163" is_pre_election: true
I20240503 14:12:13.501036 24073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cfc5cdabc784d1ea5f74b066f4e2be8" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e" is_pre_election: true
I20240503 14:12:13.501160 23812 raft_consensus.cc:2454] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 0.
I20240503 14:12:13.501243 24073 raft_consensus.cc:2454] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 0.
I20240503 14:12:13.501605 24273 leader_election.cc:302] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:13.501848 24937 raft_consensus.cc:2781] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:13.501948 24937 raft_consensus.cc:482] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:13.502036 24937 raft_consensus.cc:3037] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.503939 24937 raft_consensus.cc:504] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:13.504647 24937 leader_election.cc:288] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 election: Requested vote from peers d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473), 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675)
I20240503 14:12:13.505177 24073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cfc5cdabc784d1ea5f74b066f4e2be8" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e"
I20240503 14:12:13.505228 23812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cfc5cdabc784d1ea5f74b066f4e2be8" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163"
I20240503 14:12:13.505384 24073 raft_consensus.cc:3037] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.505405 23812 raft_consensus.cc:3037] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.508226 24073 raft_consensus.cc:2454] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 1.
I20240503 14:12:13.508235 23812 raft_consensus.cc:2454] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 1.
I20240503 14:12:13.508633 24273 leader_election.cc:302] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:13.508922 24937 raft_consensus.cc:2781] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:13.509299 24937 raft_consensus.cc:686] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [term 1 LEADER]: Becoming Leader. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Running, Role: LEADER
I20240503 14:12:13.509622 24937 consensus_queue.cc:237] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:13.512647 23538 catalog_manager.cc:5484] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff53511345114c4ab97a9a8decef3a74 (127.22.242.70). New cstate: current_term: 1 leader_uuid: "ff53511345114c4ab97a9a8decef3a74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:13.565657 25095 raft_consensus.cc:482] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:13.565871 25095 raft_consensus.cc:504] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:13.566555 25095 leader_election.cc:288] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067), 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675)
I20240503 14:12:13.567041 24073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c2d15fb0434a22afc3191ed8df3aac" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e" is_pre_election: true
I20240503 14:12:13.567015 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c2d15fb0434a22afc3191ed8df3aac" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74" is_pre_election: true
I20240503 14:12:13.567260 24337 raft_consensus.cc:2454] T e9c2d15fb0434a22afc3191ed8df3aac P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 0.
I20240503 14:12:13.567670 24073 raft_consensus.cc:2454] T e9c2d15fb0434a22afc3191ed8df3aac P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 0.
I20240503 14:12:13.567725 23747 leader_election.cc:302] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:13.567974 25095 raft_consensus.cc:2781] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:13.568125 25095 raft_consensus.cc:482] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:13.568284 25095 raft_consensus.cc:3037] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.570108 25095 raft_consensus.cc:504] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:13.570602 25095 leader_election.cc:288] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 election: Requested vote from peers ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067), 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675)
I20240503 14:12:13.571023 24073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c2d15fb0434a22afc3191ed8df3aac" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e"
I20240503 14:12:13.571038 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c2d15fb0434a22afc3191ed8df3aac" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74"
I20240503 14:12:13.571229 24337 raft_consensus.cc:3037] T e9c2d15fb0434a22afc3191ed8df3aac P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.571229 24073 raft_consensus.cc:3037] T e9c2d15fb0434a22afc3191ed8df3aac P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.573199 24073 raft_consensus.cc:2454] T e9c2d15fb0434a22afc3191ed8df3aac P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 1.
I20240503 14:12:13.573204 24337 raft_consensus.cc:2454] T e9c2d15fb0434a22afc3191ed8df3aac P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 1.
I20240503 14:12:13.573554 23749 leader_election.cc:302] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, d4eea540f3c241c1ba6290ca53769163; no voters:
I20240503 14:12:13.573810 25095 raft_consensus.cc:2781] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:13.574149 25095 raft_consensus.cc:686] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [term 1 LEADER]: Becoming Leader. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Running, Role: LEADER
I20240503 14:12:13.574447 25095 consensus_queue.cc:237] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:13.577363 23538 catalog_manager.cc:5484] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4eea540f3c241c1ba6290ca53769163 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "d4eea540f3c241c1ba6290ca53769163" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:13.709354 25020 raft_consensus.cc:482] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:13.709539 25020 raft_consensus.cc:504] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:13.710108 25020 leader_election.cc:288] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175), c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475)
I20240503 14:12:13.710523 23680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def4596ad4634265a15ce77a54833853" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" is_pre_election: true
I20240503 14:12:13.710587 24206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def4596ad4634265a15ce77a54833853" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308" is_pre_election: true
I20240503 14:12:13.710749 23680 raft_consensus.cc:2454] T def4596ad4634265a15ce77a54833853 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 0.
I20240503 14:12:13.710815 24206 raft_consensus.cc:2454] T def4596ad4634265a15ce77a54833853 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 0.
I20240503 14:12:13.711217 23878 leader_election.cc:302] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8795a3c2a7014baf9126dea0a6ce1bb4, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:13.711491 25020 raft_consensus.cc:2781] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:13.711591 25020 raft_consensus.cc:482] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:13.711692 25020 raft_consensus.cc:3037] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.714462 25020 raft_consensus.cc:504] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:13.715143 25020 leader_election.cc:288] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 election: Requested vote from peers 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175), c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475)
I20240503 14:12:13.715492 23680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def4596ad4634265a15ce77a54833853" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4"
I20240503 14:12:13.715535 24206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def4596ad4634265a15ce77a54833853" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308"
I20240503 14:12:13.715698 23680 raft_consensus.cc:3037] T def4596ad4634265a15ce77a54833853 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.715773 24206 raft_consensus.cc:3037] T def4596ad4634265a15ce77a54833853 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.718425 23680 raft_consensus.cc:2454] T def4596ad4634265a15ce77a54833853 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 1.
I20240503 14:12:13.718676 24206 raft_consensus.cc:2454] T def4596ad4634265a15ce77a54833853 P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 1.
I20240503 14:12:13.719166 23878 leader_election.cc:302] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8795a3c2a7014baf9126dea0a6ce1bb4, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:13.719460 25020 raft_consensus.cc:2781] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:13.719949 25020 raft_consensus.cc:686] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [term 1 LEADER]: Becoming Leader. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Running, Role: LEADER
I20240503 14:12:13.720286 25020 consensus_queue.cc:237] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:13.723297 23538 catalog_manager.cc:5484] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b reported cstate change: term changed from 0 to 1, leader changed from <none> to d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67). New cstate: current_term: 1 leader_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:14.079399 25122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:14.079727 25122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:12:14.083217 24937 consensus_queue.cc:1035] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:14.090684 25127 consensus_queue.cc:1035] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:14.090832 25122 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 14:12:14.141104 25095 consensus_queue.cc:1035] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:14.152376 25150 consensus_queue.cc:1035] T e9c2d15fb0434a22afc3191ed8df3aac P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:14.163935 25020 consensus_queue.cc:1035] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:14.214841 25120 consensus_queue.cc:1035] T def4596ad4634265a15ce77a54833853 P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:14.361088 23747 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.70:35067, user_credentials={real_user=slave}} blocked reactor thread for 50190.7us
INSERT report
rows total: 2000
time total: 293.533 ms
time per row: 0.146767 ms
I20240503 14:12:14.547039 23537 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33990:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:14.556358 23924 tablet_service.cc:1467] Processing CreateTablet for tablet 6684b0c4e392418cb7c9d64e4eb62c01 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a0fcc42a706747b79808a972ea900f15]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.556442 23661 tablet_service.cc:1467] Processing CreateTablet for tablet 6684b0c4e392418cb7c9d64e4eb62c01 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a0fcc42a706747b79808a972ea900f15]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.556851 23661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6684b0c4e392418cb7c9d64e4eb62c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.556866 23924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6684b0c4e392418cb7c9d64e4eb62c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.557696 24317 tablet_service.cc:1467] Processing CreateTablet for tablet a78b4e88ed1e47908a4a2beae34a3ab7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a0fcc42a706747b79808a972ea900f15]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.558010 24317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a78b4e88ed1e47908a4a2beae34a3ab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.561178 24185 tablet_service.cc:1467] Processing CreateTablet for tablet 174f75183c8f44c2a07ef326177d9d71 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a0fcc42a706747b79808a972ea900f15]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.561259 24055 tablet_service.cc:1467] Processing CreateTablet for tablet a78b4e88ed1e47908a4a2beae34a3ab7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a0fcc42a706747b79808a972ea900f15]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.561517 24186 tablet_service.cc:1467] Processing CreateTablet for tablet 6684b0c4e392418cb7c9d64e4eb62c01 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a0fcc42a706747b79808a972ea900f15]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.561618 24055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a78b4e88ed1e47908a4a2beae34a3ab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.562276 23792 tablet_service.cc:1467] Processing CreateTablet for tablet a78b4e88ed1e47908a4a2beae34a3ab7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a0fcc42a706747b79808a972ea900f15]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.561630 24185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174f75183c8f44c2a07ef326177d9d71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.562672 23792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a78b4e88ed1e47908a4a2beae34a3ab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.562681 23923 tablet_service.cc:1467] Processing CreateTablet for tablet 174f75183c8f44c2a07ef326177d9d71 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a0fcc42a706747b79808a972ea900f15]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.563058 23923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174f75183c8f44c2a07ef326177d9d71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.567005 24186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6684b0c4e392418cb7c9d64e4eb62c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.570315 23660 tablet_service.cc:1467] Processing CreateTablet for tablet 174f75183c8f44c2a07ef326177d9d71 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a0fcc42a706747b79808a972ea900f15]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.570883 23660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174f75183c8f44c2a07ef326177d9d71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.574800 25196 tablet_bootstrap.cc:492] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74: Bootstrap starting.
I20240503 14:12:14.578455 25196 tablet_bootstrap.cc:654] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.579409 25194 tablet_bootstrap.cc:492] T a78b4e88ed1e47908a4a2beae34a3ab7 P d4eea540f3c241c1ba6290ca53769163: Bootstrap starting.
I20240503 14:12:14.581841 25194 tablet_bootstrap.cc:654] T a78b4e88ed1e47908a4a2beae34a3ab7 P d4eea540f3c241c1ba6290ca53769163: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.581894 25195 tablet_bootstrap.cc:492] T 174f75183c8f44c2a07ef326177d9d71 P d40aeb06cfcc4c3a897b40290d12a15b: Bootstrap starting.
I20240503 14:12:14.585232 25195 tablet_bootstrap.cc:654] T 174f75183c8f44c2a07ef326177d9d71 P d40aeb06cfcc4c3a897b40290d12a15b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.590427 25194 tablet_bootstrap.cc:492] T a78b4e88ed1e47908a4a2beae34a3ab7 P d4eea540f3c241c1ba6290ca53769163: No bootstrap required, opened a new log
I20240503 14:12:14.590677 25194 ts_tablet_manager.cc:1367] T a78b4e88ed1e47908a4a2beae34a3ab7 P d4eea540f3c241c1ba6290ca53769163: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240503 14:12:14.591542 25194 raft_consensus.cc:348] T a78b4e88ed1e47908a4a2beae34a3ab7 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:14.591904 25194 raft_consensus.cc:374] T a78b4e88ed1e47908a4a2beae34a3ab7 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.592026 25194 raft_consensus.cc:729] T a78b4e88ed1e47908a4a2beae34a3ab7 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.592516 25194 consensus_queue.cc:260] T a78b4e88ed1e47908a4a2beae34a3ab7 P d4eea540f3c241c1ba6290ca53769163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:14.593196 25194 ts_tablet_manager.cc:1398] T a78b4e88ed1e47908a4a2beae34a3ab7 P d4eea540f3c241c1ba6290ca53769163: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:14.594446 25196 tablet_bootstrap.cc:492] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74: No bootstrap required, opened a new log
I20240503 14:12:14.594687 25196 ts_tablet_manager.cc:1367] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.009s
I20240503 14:12:14.595793 25196 raft_consensus.cc:348] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:14.596086 25196 raft_consensus.cc:374] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.596241 25196 raft_consensus.cc:729] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.596530 25196 consensus_queue.cc:260] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:14.597249 25196 ts_tablet_manager.cc:1398] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:14.600478 25201 tablet_bootstrap.cc:492] T 6684b0c4e392418cb7c9d64e4eb62c01 P 8795a3c2a7014baf9126dea0a6ce1bb4: Bootstrap starting.
I20240503 14:12:14.602766 25201 tablet_bootstrap.cc:654] T 6684b0c4e392418cb7c9d64e4eb62c01 P 8795a3c2a7014baf9126dea0a6ce1bb4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.602919 25202 tablet_bootstrap.cc:492] T 6684b0c4e392418cb7c9d64e4eb62c01 P c9a573f6faf64131b114af1e33c1b308: Bootstrap starting.
I20240503 14:12:14.604745 25202 tablet_bootstrap.cc:654] T 6684b0c4e392418cb7c9d64e4eb62c01 P c9a573f6faf64131b114af1e33c1b308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.611249 25201 tablet_bootstrap.cc:492] T 6684b0c4e392418cb7c9d64e4eb62c01 P 8795a3c2a7014baf9126dea0a6ce1bb4: No bootstrap required, opened a new log
I20240503 14:12:14.611531 25201 ts_tablet_manager.cc:1367] T 6684b0c4e392418cb7c9d64e4eb62c01 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.004s
I20240503 14:12:14.612560 25201 raft_consensus.cc:348] T 6684b0c4e392418cb7c9d64e4eb62c01 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:14.612794 25197 tablet_bootstrap.cc:492] T a78b4e88ed1e47908a4a2beae34a3ab7 P 1e79cc5c647f476c98101f9e48b22b9e: Bootstrap starting.
I20240503 14:12:14.612946 25202 tablet_bootstrap.cc:492] T 6684b0c4e392418cb7c9d64e4eb62c01 P c9a573f6faf64131b114af1e33c1b308: No bootstrap required, opened a new log
I20240503 14:12:14.612977 25201 raft_consensus.cc:374] T 6684b0c4e392418cb7c9d64e4eb62c01 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.613026 25195 tablet_bootstrap.cc:492] T 174f75183c8f44c2a07ef326177d9d71 P d40aeb06cfcc4c3a897b40290d12a15b: No bootstrap required, opened a new log
I20240503 14:12:14.613153 25202 ts_tablet_manager.cc:1367] T 6684b0c4e392418cb7c9d64e4eb62c01 P c9a573f6faf64131b114af1e33c1b308: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20240503 14:12:14.613152 25201 raft_consensus.cc:729] T 6684b0c4e392418cb7c9d64e4eb62c01 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.613185 25195 ts_tablet_manager.cc:1367] T 174f75183c8f44c2a07ef326177d9d71 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.009s
I20240503 14:12:14.613420 25201 consensus_queue.cc:260] T 6684b0c4e392418cb7c9d64e4eb62c01 P 8795a3c2a7014baf9126dea0a6ce1bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:14.613955 25195 raft_consensus.cc:348] T 174f75183c8f44c2a07ef326177d9d71 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:14.614039 25202 raft_consensus.cc:348] T 6684b0c4e392418cb7c9d64e4eb62c01 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:14.614261 25202 raft_consensus.cc:374] T 6684b0c4e392418cb7c9d64e4eb62c01 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.614353 25202 raft_consensus.cc:729] T 6684b0c4e392418cb7c9d64e4eb62c01 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.614629 25197 tablet_bootstrap.cc:654] T a78b4e88ed1e47908a4a2beae34a3ab7 P 1e79cc5c647f476c98101f9e48b22b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.614591 25202 consensus_queue.cc:260] T 6684b0c4e392418cb7c9d64e4eb62c01 P c9a573f6faf64131b114af1e33c1b308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:14.614912 25201 ts_tablet_manager.cc:1398] T 6684b0c4e392418cb7c9d64e4eb62c01 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:14.615368 25202 ts_tablet_manager.cc:1398] T 6684b0c4e392418cb7c9d64e4eb62c01 P c9a573f6faf64131b114af1e33c1b308: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:14.615820 25202 tablet_bootstrap.cc:492] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308: Bootstrap starting.
I20240503 14:12:14.616075 25201 tablet_bootstrap.cc:492] T 174f75183c8f44c2a07ef326177d9d71 P 8795a3c2a7014baf9126dea0a6ce1bb4: Bootstrap starting.
I20240503 14:12:14.617379 25195 raft_consensus.cc:374] T 174f75183c8f44c2a07ef326177d9d71 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.617509 25202 tablet_bootstrap.cc:654] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.618466 25195 raft_consensus.cc:729] T 174f75183c8f44c2a07ef326177d9d71 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.618880 25195 consensus_queue.cc:260] T 174f75183c8f44c2a07ef326177d9d71 P d40aeb06cfcc4c3a897b40290d12a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:14.619889 25195 ts_tablet_manager.cc:1398] T 174f75183c8f44c2a07ef326177d9d71 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20240503 14:12:14.620380 25195 tablet_bootstrap.cc:492] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b: Bootstrap starting.
I20240503 14:12:14.622184 25195 tablet_bootstrap.cc:654] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.622828 25202 tablet_bootstrap.cc:492] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308: No bootstrap required, opened a new log
I20240503 14:12:14.623044 25202 ts_tablet_manager.cc:1367] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20240503 14:12:14.623651 25201 tablet_bootstrap.cc:654] T 174f75183c8f44c2a07ef326177d9d71 P 8795a3c2a7014baf9126dea0a6ce1bb4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.623853 25202 raft_consensus.cc:348] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:14.624086 25202 raft_consensus.cc:374] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.624198 25202 raft_consensus.cc:729] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.624541 25202 consensus_queue.cc:260] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:14.626127 25202 ts_tablet_manager.cc:1398] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 14:12:14.628266 25195 tablet_bootstrap.cc:492] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b: No bootstrap required, opened a new log
I20240503 14:12:14.628312 25197 tablet_bootstrap.cc:492] T a78b4e88ed1e47908a4a2beae34a3ab7 P 1e79cc5c647f476c98101f9e48b22b9e: No bootstrap required, opened a new log
I20240503 14:12:14.628464 25195 ts_tablet_manager.cc:1367] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240503 14:12:14.629462 25195 raft_consensus.cc:348] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:14.629731 25195 raft_consensus.cc:374] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.629817 25195 raft_consensus.cc:729] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.630223 25197 ts_tablet_manager.cc:1367] T a78b4e88ed1e47908a4a2beae34a3ab7 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20240503 14:12:14.630066 25195 consensus_queue.cc:260] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:14.630806 25195 ts_tablet_manager.cc:1398] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:14.630980 25197 raft_consensus.cc:348] T a78b4e88ed1e47908a4a2beae34a3ab7 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:14.631166 25197 raft_consensus.cc:374] T a78b4e88ed1e47908a4a2beae34a3ab7 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.631276 25197 raft_consensus.cc:729] T a78b4e88ed1e47908a4a2beae34a3ab7 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.631489 25197 consensus_queue.cc:260] T a78b4e88ed1e47908a4a2beae34a3ab7 P 1e79cc5c647f476c98101f9e48b22b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:14.632076 25197 ts_tablet_manager.cc:1398] T a78b4e88ed1e47908a4a2beae34a3ab7 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:14.633356 25201 tablet_bootstrap.cc:492] T 174f75183c8f44c2a07ef326177d9d71 P 8795a3c2a7014baf9126dea0a6ce1bb4: No bootstrap required, opened a new log
I20240503 14:12:14.633533 25201 ts_tablet_manager.cc:1367] T 174f75183c8f44c2a07ef326177d9d71 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.003s
I20240503 14:12:14.634487 25201 raft_consensus.cc:348] T 174f75183c8f44c2a07ef326177d9d71 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:14.634667 25201 raft_consensus.cc:374] T 174f75183c8f44c2a07ef326177d9d71 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.634752 25201 raft_consensus.cc:729] T 174f75183c8f44c2a07ef326177d9d71 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.635035 25201 consensus_queue.cc:260] T 174f75183c8f44c2a07ef326177d9d71 P 8795a3c2a7014baf9126dea0a6ce1bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:14.635507 25201 ts_tablet_manager.cc:1398] T 174f75183c8f44c2a07ef326177d9d71 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:14.650242 25176 raft_consensus.cc:482] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:14.650427 25176 raft_consensus.cc:504] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:14.651177 25176 leader_election.cc:288] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175), c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475)
I20240503 14:12:14.651674 23681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6684b0c4e392418cb7c9d64e4eb62c01" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" is_pre_election: true
I20240503 14:12:14.651780 24206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6684b0c4e392418cb7c9d64e4eb62c01" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308" is_pre_election: true
I20240503 14:12:14.651932 23681 raft_consensus.cc:2454] T 6684b0c4e392418cb7c9d64e4eb62c01 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 0.
I20240503 14:12:14.652017 24206 raft_consensus.cc:2454] T 6684b0c4e392418cb7c9d64e4eb62c01 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 0.
I20240503 14:12:14.652429 23878 leader_election.cc:302] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8795a3c2a7014baf9126dea0a6ce1bb4, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:14.652738 25176 raft_consensus.cc:2781] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:14.652848 25176 raft_consensus.cc:482] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:14.652938 25176 raft_consensus.cc:3037] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:14.654827 25176 raft_consensus.cc:504] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:14.655342 25176 leader_election.cc:288] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 election: Requested vote from peers 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175), c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475)
I20240503 14:12:14.655818 23681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6684b0c4e392418cb7c9d64e4eb62c01" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4"
I20240503 14:12:14.655876 24206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6684b0c4e392418cb7c9d64e4eb62c01" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308"
I20240503 14:12:14.656001 23681 raft_consensus.cc:3037] T 6684b0c4e392418cb7c9d64e4eb62c01 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:14.656072 24206 raft_consensus.cc:3037] T 6684b0c4e392418cb7c9d64e4eb62c01 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:14.658082 23681 raft_consensus.cc:2454] T 6684b0c4e392418cb7c9d64e4eb62c01 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 1.
I20240503 14:12:14.658092 24206 raft_consensus.cc:2454] T 6684b0c4e392418cb7c9d64e4eb62c01 P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 1.
I20240503 14:12:14.658460 23878 leader_election.cc:302] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:14.659130 25176 raft_consensus.cc:2781] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:14.659399 25176 raft_consensus.cc:686] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [term 1 LEADER]: Becoming Leader. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Running, Role: LEADER
I20240503 14:12:14.659739 25176 consensus_queue.cc:237] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:14.662559 23537 catalog_manager.cc:5484] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b reported cstate change: term changed from 0 to 1, leader changed from <none> to d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67). New cstate: current_term: 1 leader_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:14.669776 25127 raft_consensus.cc:482] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:14.669932 25127 raft_consensus.cc:504] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:14.670491 25127 leader_election.cc:288] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675), d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473)
I20240503 14:12:14.671013 24074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78b4e88ed1e47908a4a2beae34a3ab7" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e" is_pre_election: true
I20240503 14:12:14.671036 23812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78b4e88ed1e47908a4a2beae34a3ab7" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163" is_pre_election: true
I20240503 14:12:14.671218 24074 raft_consensus.cc:2454] T a78b4e88ed1e47908a4a2beae34a3ab7 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 0.
I20240503 14:12:14.671233 23812 raft_consensus.cc:2454] T a78b4e88ed1e47908a4a2beae34a3ab7 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 0.
I20240503 14:12:14.671630 24273 leader_election.cc:302] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:14.671900 25127 raft_consensus.cc:2781] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:14.671986 25127 raft_consensus.cc:482] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:14.672089 25127 raft_consensus.cc:3037] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:14.673894 25127 raft_consensus.cc:504] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:14.674466 25127 leader_election.cc:288] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 election: Requested vote from peers 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675), d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473)
I20240503 14:12:14.674844 24074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78b4e88ed1e47908a4a2beae34a3ab7" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e"
I20240503 14:12:14.675004 24074 raft_consensus.cc:3037] T a78b4e88ed1e47908a4a2beae34a3ab7 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:14.674926 23812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78b4e88ed1e47908a4a2beae34a3ab7" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163"
I20240503 14:12:14.675144 23812 raft_consensus.cc:3037] T a78b4e88ed1e47908a4a2beae34a3ab7 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:14.676956 24074 raft_consensus.cc:2454] T a78b4e88ed1e47908a4a2beae34a3ab7 P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 1.
I20240503 14:12:14.677016 23812 raft_consensus.cc:2454] T a78b4e88ed1e47908a4a2beae34a3ab7 P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 1.
I20240503 14:12:14.677325 24273 leader_election.cc:302] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:14.677569 25127 raft_consensus.cc:2781] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:14.677742 25127 raft_consensus.cc:686] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [term 1 LEADER]: Becoming Leader. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Running, Role: LEADER
I20240503 14:12:14.677994 25127 consensus_queue.cc:237] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } }
I20240503 14:12:14.680853 23537 catalog_manager.cc:5484] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff53511345114c4ab97a9a8decef3a74 (127.22.242.70). New cstate: current_term: 1 leader_uuid: "ff53511345114c4ab97a9a8decef3a74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:14.844246 25082 raft_consensus.cc:482] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:14.844425 25082 raft_consensus.cc:504] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:14.845189 25082 leader_election.cc:288] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697), 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175)
I20240503 14:12:14.845561 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174f75183c8f44c2a07ef326177d9d71" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" is_pre_election: true
I20240503 14:12:14.845775 23944 raft_consensus.cc:2454] T 174f75183c8f44c2a07ef326177d9d71 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 0.
I20240503 14:12:14.845741 23681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174f75183c8f44c2a07ef326177d9d71" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" is_pre_election: true
I20240503 14:12:14.845947 23681 raft_consensus.cc:2454] T 174f75183c8f44c2a07ef326177d9d71 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 0.
I20240503 14:12:14.846253 24139 leader_election.cc:302] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:14.846552 25082 raft_consensus.cc:2781] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:14.846655 25082 raft_consensus.cc:482] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:14.846750 25082 raft_consensus.cc:3037] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:14.848745 25082 raft_consensus.cc:504] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:14.849283 25082 leader_election.cc:288] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 election: Requested vote from peers d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697), 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175)
I20240503 14:12:14.849772 23681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174f75183c8f44c2a07ef326177d9d71" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4"
I20240503 14:12:14.849772 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174f75183c8f44c2a07ef326177d9d71" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b"
I20240503 14:12:14.849969 23944 raft_consensus.cc:3037] T 174f75183c8f44c2a07ef326177d9d71 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:14.849969 23681 raft_consensus.cc:3037] T 174f75183c8f44c2a07ef326177d9d71 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:14.852802 23681 raft_consensus.cc:2454] T 174f75183c8f44c2a07ef326177d9d71 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 1.
I20240503 14:12:14.852802 23944 raft_consensus.cc:2454] T 174f75183c8f44c2a07ef326177d9d71 P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 1.
I20240503 14:12:14.853206 24139 leader_election.cc:302] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:14.853523 25082 raft_consensus.cc:2781] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:14.853909 25082 raft_consensus.cc:686] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [term 1 LEADER]: Becoming Leader. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Running, Role: LEADER
I20240503 14:12:14.854224 25082 consensus_queue.cc:237] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:14.857539 23537 catalog_manager.cc:5484] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9a573f6faf64131b114af1e33c1b308 (127.22.242.69). New cstate: current_term: 1 leader_uuid: "c9a573f6faf64131b114af1e33c1b308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:15.162575 25176 consensus_queue.cc:1035] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:15.168722 25215 consensus_queue.cc:1035] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:15.176366 25188 consensus_queue.cc:1035] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:15.216259 25188 consensus_queue.cc:1035] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:15.258102 25213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:15.258498 25213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:15.267628 25213 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 14:12:15.404297 23681 raft_consensus.cc:1261] T 174f75183c8f44c2a07ef326177d9d71 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Refusing update from remote peer c9a573f6faf64131b114af1e33c1b308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:15.404347 23944 raft_consensus.cc:1261] T 174f75183c8f44c2a07ef326177d9d71 P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Refusing update from remote peer c9a573f6faf64131b114af1e33c1b308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:15.405184 25082 consensus_queue.cc:1035] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:15.405843 25082 consensus_queue.cc:1035] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:15.450569 25221 mvcc.cc:205] Tried to move back new op lower bound from 7023597712998498304 to 7023597710765580288. Current Snapshot: MvccSnapshot[applied={T|T < 7023597712998498304 or (T in {7023597712998498304})}]
INSERT report
rows total: 2000
time total: 279.679 ms
time per row: 0.13984 ms
I20240503 14:12:15.695298 23538 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33990:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:15.705354 23792 tablet_service.cc:1467] Processing CreateTablet for tablet 630ea99d845540beb727a686457d0e18 (DEFAULT_TABLE table=rebalance_test_table_9 [id=649cfaa5d09a453d9589867ffa4b93f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.705514 24055 tablet_service.cc:1467] Processing CreateTablet for tablet 630ea99d845540beb727a686457d0e18 (DEFAULT_TABLE table=rebalance_test_table_9 [id=649cfaa5d09a453d9589867ffa4b93f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.705827 23792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 630ea99d845540beb727a686457d0e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.705968 24055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 630ea99d845540beb727a686457d0e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.706719 24317 tablet_service.cc:1467] Processing CreateTablet for tablet 630ea99d845540beb727a686457d0e18 (DEFAULT_TABLE table=rebalance_test_table_9 [id=649cfaa5d09a453d9589867ffa4b93f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.707100 24317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 630ea99d845540beb727a686457d0e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.707376 23923 tablet_service.cc:1467] Processing CreateTablet for tablet 32781a4a39434a96aa36a9d896a58786 (DEFAULT_TABLE table=rebalance_test_table_9 [id=649cfaa5d09a453d9589867ffa4b93f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.707764 23923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32781a4a39434a96aa36a9d896a58786. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.708844 23793 tablet_service.cc:1467] Processing CreateTablet for tablet c306f7c2c4d7493990bdf5242238a381 (DEFAULT_TABLE table=rebalance_test_table_9 [id=649cfaa5d09a453d9589867ffa4b93f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.709231 23793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c306f7c2c4d7493990bdf5242238a381. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.713347 23660 tablet_service.cc:1467] Processing CreateTablet for tablet 32781a4a39434a96aa36a9d896a58786 (DEFAULT_TABLE table=rebalance_test_table_9 [id=649cfaa5d09a453d9589867ffa4b93f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.713354 24316 tablet_service.cc:1467] Processing CreateTablet for tablet c306f7c2c4d7493990bdf5242238a381 (DEFAULT_TABLE table=rebalance_test_table_9 [id=649cfaa5d09a453d9589867ffa4b93f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.713788 24316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c306f7c2c4d7493990bdf5242238a381. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.713840 23660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32781a4a39434a96aa36a9d896a58786. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.716614 24186 tablet_service.cc:1467] Processing CreateTablet for tablet 32781a4a39434a96aa36a9d896a58786 (DEFAULT_TABLE table=rebalance_test_table_9 [id=649cfaa5d09a453d9589867ffa4b93f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.717053 24186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32781a4a39434a96aa36a9d896a58786. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.721860 24054 tablet_service.cc:1467] Processing CreateTablet for tablet c306f7c2c4d7493990bdf5242238a381 (DEFAULT_TABLE table=rebalance_test_table_9 [id=649cfaa5d09a453d9589867ffa4b93f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.722298 24054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c306f7c2c4d7493990bdf5242238a381. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.725581 25286 tablet_bootstrap.cc:492] T 630ea99d845540beb727a686457d0e18 P ff53511345114c4ab97a9a8decef3a74: Bootstrap starting.
I20240503 14:12:15.727345 25286 tablet_bootstrap.cc:654] T 630ea99d845540beb727a686457d0e18 P ff53511345114c4ab97a9a8decef3a74: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.743633 25286 tablet_bootstrap.cc:492] T 630ea99d845540beb727a686457d0e18 P ff53511345114c4ab97a9a8decef3a74: No bootstrap required, opened a new log
I20240503 14:12:15.744225 25286 ts_tablet_manager.cc:1367] T 630ea99d845540beb727a686457d0e18 P ff53511345114c4ab97a9a8decef3a74: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20240503 14:12:15.745688 25286 raft_consensus.cc:348] T 630ea99d845540beb727a686457d0e18 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:15.746776 25287 tablet_bootstrap.cc:492] T 32781a4a39434a96aa36a9d896a58786 P d40aeb06cfcc4c3a897b40290d12a15b: Bootstrap starting.
I20240503 14:12:15.747642 25286 raft_consensus.cc:374] T 630ea99d845540beb727a686457d0e18 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.747927 25286 raft_consensus.cc:729] T 630ea99d845540beb727a686457d0e18 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.748368 25286 consensus_queue.cc:260] T 630ea99d845540beb727a686457d0e18 P ff53511345114c4ab97a9a8decef3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:15.748871 25287 tablet_bootstrap.cc:654] T 32781a4a39434a96aa36a9d896a58786 P d40aeb06cfcc4c3a897b40290d12a15b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.748912 25286 ts_tablet_manager.cc:1398] T 630ea99d845540beb727a686457d0e18 P ff53511345114c4ab97a9a8decef3a74: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:15.750775 25286 tablet_bootstrap.cc:492] T c306f7c2c4d7493990bdf5242238a381 P ff53511345114c4ab97a9a8decef3a74: Bootstrap starting.
I20240503 14:12:15.752743 25286 tablet_bootstrap.cc:654] T c306f7c2c4d7493990bdf5242238a381 P ff53511345114c4ab97a9a8decef3a74: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.756589 25287 tablet_bootstrap.cc:492] T 32781a4a39434a96aa36a9d896a58786 P d40aeb06cfcc4c3a897b40290d12a15b: No bootstrap required, opened a new log
I20240503 14:12:15.756827 25287 ts_tablet_manager.cc:1367] T 32781a4a39434a96aa36a9d896a58786 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20240503 14:12:15.757712 25287 raft_consensus.cc:348] T 32781a4a39434a96aa36a9d896a58786 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:15.758258 25287 raft_consensus.cc:374] T 32781a4a39434a96aa36a9d896a58786 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.758383 25287 raft_consensus.cc:729] T 32781a4a39434a96aa36a9d896a58786 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.758628 25287 consensus_queue.cc:260] T 32781a4a39434a96aa36a9d896a58786 P d40aeb06cfcc4c3a897b40290d12a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:15.759227 25287 ts_tablet_manager.cc:1398] T 32781a4a39434a96aa36a9d896a58786 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:12:15.770689 25292 tablet_bootstrap.cc:492] T c306f7c2c4d7493990bdf5242238a381 P 1e79cc5c647f476c98101f9e48b22b9e: Bootstrap starting.
I20240503 14:12:15.770956 25293 tablet_bootstrap.cc:492] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4: Bootstrap starting.
I20240503 14:12:15.773198 25289 tablet_bootstrap.cc:492] T 32781a4a39434a96aa36a9d896a58786 P c9a573f6faf64131b114af1e33c1b308: Bootstrap starting.
I20240503 14:12:15.775532 25293 tablet_bootstrap.cc:654] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.777336 25292 tablet_bootstrap.cc:654] T c306f7c2c4d7493990bdf5242238a381 P 1e79cc5c647f476c98101f9e48b22b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.777809 25289 tablet_bootstrap.cc:654] T 32781a4a39434a96aa36a9d896a58786 P c9a573f6faf64131b114af1e33c1b308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.781559 25286 tablet_bootstrap.cc:492] T c306f7c2c4d7493990bdf5242238a381 P ff53511345114c4ab97a9a8decef3a74: No bootstrap required, opened a new log
I20240503 14:12:15.781606 25285 tablet_bootstrap.cc:492] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163: Bootstrap starting.
I20240503 14:12:15.781736 25286 ts_tablet_manager.cc:1367] T c306f7c2c4d7493990bdf5242238a381 P ff53511345114c4ab97a9a8decef3a74: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.011s
I20240503 14:12:15.782449 25286 raft_consensus.cc:348] T c306f7c2c4d7493990bdf5242238a381 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:15.782636 25286 raft_consensus.cc:374] T c306f7c2c4d7493990bdf5242238a381 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.782747 25286 raft_consensus.cc:729] T c306f7c2c4d7493990bdf5242238a381 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.783002 25286 consensus_queue.cc:260] T c306f7c2c4d7493990bdf5242238a381 P ff53511345114c4ab97a9a8decef3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:15.783596 25286 ts_tablet_manager.cc:1398] T c306f7c2c4d7493990bdf5242238a381 P ff53511345114c4ab97a9a8decef3a74: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:15.785037 25289 tablet_bootstrap.cc:492] T 32781a4a39434a96aa36a9d896a58786 P c9a573f6faf64131b114af1e33c1b308: No bootstrap required, opened a new log
I20240503 14:12:15.785245 25289 ts_tablet_manager.cc:1367] T 32781a4a39434a96aa36a9d896a58786 P c9a573f6faf64131b114af1e33c1b308: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240503 14:12:15.786063 25289 raft_consensus.cc:348] T 32781a4a39434a96aa36a9d896a58786 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:15.786315 25289 raft_consensus.cc:374] T 32781a4a39434a96aa36a9d896a58786 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.786434 25289 raft_consensus.cc:729] T 32781a4a39434a96aa36a9d896a58786 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.786701 25289 consensus_queue.cc:260] T 32781a4a39434a96aa36a9d896a58786 P c9a573f6faf64131b114af1e33c1b308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:15.788184 25289 ts_tablet_manager.cc:1398] T 32781a4a39434a96aa36a9d896a58786 P c9a573f6faf64131b114af1e33c1b308: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:15.789181 25285 tablet_bootstrap.cc:654] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.791837 25292 tablet_bootstrap.cc:492] T c306f7c2c4d7493990bdf5242238a381 P 1e79cc5c647f476c98101f9e48b22b9e: No bootstrap required, opened a new log
I20240503 14:12:15.792053 25292 ts_tablet_manager.cc:1367] T c306f7c2c4d7493990bdf5242238a381 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.007s
I20240503 14:12:15.792536 25285 tablet_bootstrap.cc:492] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163: No bootstrap required, opened a new log
I20240503 14:12:15.792737 25285 ts_tablet_manager.cc:1367] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 14:12:15.792902 25292 raft_consensus.cc:348] T c306f7c2c4d7493990bdf5242238a381 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:15.793109 25292 raft_consensus.cc:374] T c306f7c2c4d7493990bdf5242238a381 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.793227 25292 raft_consensus.cc:729] T c306f7c2c4d7493990bdf5242238a381 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.793473 25292 consensus_queue.cc:260] T c306f7c2c4d7493990bdf5242238a381 P 1e79cc5c647f476c98101f9e48b22b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:15.793612 25285 raft_consensus.cc:348] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:15.793799 25285 raft_consensus.cc:374] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.793896 25285 raft_consensus.cc:729] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.794015 25292 ts_tablet_manager.cc:1398] T c306f7c2c4d7493990bdf5242238a381 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:15.794144 25285 consensus_queue.cc:260] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:15.794646 25285 ts_tablet_manager.cc:1398] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:15.795176 25293 tablet_bootstrap.cc:492] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4: No bootstrap required, opened a new log
I20240503 14:12:15.795418 25293 ts_tablet_manager.cc:1367] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.000s
I20240503 14:12:15.796847 25285 tablet_bootstrap.cc:492] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163: Bootstrap starting.
I20240503 14:12:15.796277 25293 raft_consensus.cc:348] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:15.797348 25293 raft_consensus.cc:374] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.797497 25293 raft_consensus.cc:729] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.797809 25293 consensus_queue.cc:260] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:15.798393 25293 ts_tablet_manager.cc:1398] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:12:15.798599 25285 tablet_bootstrap.cc:654] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.800773 25292 tablet_bootstrap.cc:492] T 630ea99d845540beb727a686457d0e18 P 1e79cc5c647f476c98101f9e48b22b9e: Bootstrap starting.
I20240503 14:12:15.803308 25292 tablet_bootstrap.cc:654] T 630ea99d845540beb727a686457d0e18 P 1e79cc5c647f476c98101f9e48b22b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.807443 25292 tablet_bootstrap.cc:492] T 630ea99d845540beb727a686457d0e18 P 1e79cc5c647f476c98101f9e48b22b9e: No bootstrap required, opened a new log
I20240503 14:12:15.807618 25292 ts_tablet_manager.cc:1367] T 630ea99d845540beb727a686457d0e18 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240503 14:12:15.808434 25292 raft_consensus.cc:348] T 630ea99d845540beb727a686457d0e18 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:15.808640 25292 raft_consensus.cc:374] T 630ea99d845540beb727a686457d0e18 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.808724 25292 raft_consensus.cc:729] T 630ea99d845540beb727a686457d0e18 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.808977 25292 consensus_queue.cc:260] T 630ea99d845540beb727a686457d0e18 P 1e79cc5c647f476c98101f9e48b22b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:15.809453 25292 ts_tablet_manager.cc:1398] T 630ea99d845540beb727a686457d0e18 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:15.824712 25285 tablet_bootstrap.cc:492] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163: No bootstrap required, opened a new log
I20240503 14:12:15.824909 25285 ts_tablet_manager.cc:1367] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.005s
I20240503 14:12:15.825592 25285 raft_consensus.cc:348] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:15.825788 25285 raft_consensus.cc:374] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.825865 25285 raft_consensus.cc:729] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.826114 25285 consensus_queue.cc:260] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:15.826598 25285 ts_tablet_manager.cc:1398] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:15.827227 25174 raft_consensus.cc:482] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:15.827350 25174 raft_consensus.cc:504] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:15.828121 25174 leader_election.cc:288] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067), 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675)
I20240503 14:12:15.828600 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630ea99d845540beb727a686457d0e18" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74" is_pre_election: true
I20240503 14:12:15.828716 24074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630ea99d845540beb727a686457d0e18" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e" is_pre_election: true
I20240503 14:12:15.828886 24337 raft_consensus.cc:2454] T 630ea99d845540beb727a686457d0e18 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 0.
I20240503 14:12:15.828944 24074 raft_consensus.cc:2454] T 630ea99d845540beb727a686457d0e18 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 0.
I20240503 14:12:15.829497 23749 leader_election.cc:302] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, d4eea540f3c241c1ba6290ca53769163; no voters:
I20240503 14:12:15.829878 25174 raft_consensus.cc:2781] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:15.830010 25174 raft_consensus.cc:482] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:15.830099 25174 raft_consensus.cc:3037] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.832760 25174 raft_consensus.cc:504] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:15.833293 25174 leader_election.cc:288] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 election: Requested vote from peers ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067), 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675)
I20240503 14:12:15.833634 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630ea99d845540beb727a686457d0e18" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74"
I20240503 14:12:15.833846 24337 raft_consensus.cc:3037] T 630ea99d845540beb727a686457d0e18 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.833945 24074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630ea99d845540beb727a686457d0e18" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e"
I20240503 14:12:15.834151 24074 raft_consensus.cc:3037] T 630ea99d845540beb727a686457d0e18 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.836666 24337 raft_consensus.cc:2454] T 630ea99d845540beb727a686457d0e18 P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 1.
I20240503 14:12:15.836903 24074 raft_consensus.cc:2454] T 630ea99d845540beb727a686457d0e18 P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 1.
I20240503 14:12:15.837111 23747 leader_election.cc:302] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:15.837443 25174 raft_consensus.cc:2781] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:15.837575 25174 raft_consensus.cc:686] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [term 1 LEADER]: Becoming Leader. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Running, Role: LEADER
I20240503 14:12:15.837841 25174 consensus_queue.cc:237] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } }
I20240503 14:12:15.840894 23538 catalog_manager.cc:5484] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4eea540f3c241c1ba6290ca53769163 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "d4eea540f3c241c1ba6290ca53769163" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:15.856245 25250 raft_consensus.cc:482] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:15.856438 25108 raft_consensus.cc:482] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:15.856424 25250 raft_consensus.cc:504] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:15.856624 25108 raft_consensus.cc:504] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:15.857164 25250 leader_election.cc:288] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067), 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675)
I20240503 14:12:15.857306 25108 leader_election.cc:288] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475), d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697)
I20240503 14:12:15.857704 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c306f7c2c4d7493990bdf5242238a381" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74" is_pre_election: true
I20240503 14:12:15.857833 24206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32781a4a39434a96aa36a9d896a58786" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308" is_pre_election: true
I20240503 14:12:15.857812 23943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32781a4a39434a96aa36a9d896a58786" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" is_pre_election: true
I20240503 14:12:15.857888 24074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c306f7c2c4d7493990bdf5242238a381" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e" is_pre_election: true
I20240503 14:12:15.857993 24337 raft_consensus.cc:2454] T c306f7c2c4d7493990bdf5242238a381 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 0.
I20240503 14:12:15.858057 24206 raft_consensus.cc:2454] T 32781a4a39434a96aa36a9d896a58786 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 0.
I20240503 14:12:15.858062 23943 raft_consensus.cc:2454] T 32781a4a39434a96aa36a9d896a58786 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 0.
I20240503 14:12:15.858074 24074 raft_consensus.cc:2454] T c306f7c2c4d7493990bdf5242238a381 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 0.
I20240503 14:12:15.858399 23747 leader_election.cc:302] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:15.858502 23615 leader_election.cc:302] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8795a3c2a7014baf9126dea0a6ce1bb4, c9a573f6faf64131b114af1e33c1b308; no voters:
I20240503 14:12:15.858750 25250 raft_consensus.cc:2781] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:15.858789 25108 raft_consensus.cc:2781] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:15.858865 25250 raft_consensus.cc:482] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:15.858928 25108 raft_consensus.cc:482] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:15.858983 25250 raft_consensus.cc:3037] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.859030 25108 raft_consensus.cc:3037] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.861351 25250 raft_consensus.cc:504] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:15.861420 25108 raft_consensus.cc:504] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:15.862039 25108 leader_election.cc:288] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 election: Requested vote from peers c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475), d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697)
I20240503 14:12:15.862272 25250 leader_election.cc:288] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 election: Requested vote from peers ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067), 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675)
I20240503 14:12:15.862377 24206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32781a4a39434a96aa36a9d896a58786" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308"
I20240503 14:12:15.862598 24206 raft_consensus.cc:3037] T 32781a4a39434a96aa36a9d896a58786 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.862605 24074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c306f7c2c4d7493990bdf5242238a381" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e"
I20240503 14:12:15.862600 23943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32781a4a39434a96aa36a9d896a58786" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b"
I20240503 14:12:15.862766 24074 raft_consensus.cc:3037] T c306f7c2c4d7493990bdf5242238a381 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.862792 23943 raft_consensus.cc:3037] T 32781a4a39434a96aa36a9d896a58786 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.862563 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c306f7c2c4d7493990bdf5242238a381" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74"
I20240503 14:12:15.863607 24337 raft_consensus.cc:3037] T c306f7c2c4d7493990bdf5242238a381 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.865355 24206 raft_consensus.cc:2454] T 32781a4a39434a96aa36a9d896a58786 P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 1.
I20240503 14:12:15.865386 23943 raft_consensus.cc:2454] T 32781a4a39434a96aa36a9d896a58786 P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 1.
I20240503 14:12:15.865710 23614 leader_election.cc:302] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8795a3c2a7014baf9126dea0a6ce1bb4, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:15.865552 24074 raft_consensus.cc:2454] T c306f7c2c4d7493990bdf5242238a381 P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 1.
I20240503 14:12:15.866267 24337 raft_consensus.cc:2454] T c306f7c2c4d7493990bdf5242238a381 P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 1.
I20240503 14:12:15.866391 25108 raft_consensus.cc:2781] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:15.866518 23749 leader_election.cc:302] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, d4eea540f3c241c1ba6290ca53769163; no voters:
I20240503 14:12:15.866562 25108 raft_consensus.cc:686] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 LEADER]: Becoming Leader. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Running, Role: LEADER
I20240503 14:12:15.866979 25250 raft_consensus.cc:2781] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:15.866932 25108 consensus_queue.cc:237] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:15.868999 25250 raft_consensus.cc:686] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [term 1 LEADER]: Becoming Leader. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Running, Role: LEADER
I20240503 14:12:15.869308 25250 consensus_queue.cc:237] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:15.869901 23538 catalog_manager.cc:5484] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:15.872125 23534 catalog_manager.cc:5484] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4eea540f3c241c1ba6290ca53769163 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "d4eea540f3c241c1ba6290ca53769163" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } health_report { overall_health: HEALTHY } } }
W20240503 14:12:16.240859 25305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:16.241269 25305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:12:16.248777 25250 consensus_queue.cc:1035] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:16.251914 25305 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:12:16.257081 25250 consensus_queue.cc:1035] T c306f7c2c4d7493990bdf5242238a381 P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240503 14:12:16.312685 25301 consensus_queue.cc:1035] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:16.333067 25301 consensus_queue.cc:1035] T 32781a4a39434a96aa36a9d896a58786 P 8795a3c2a7014baf9126dea0a6ce1bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:16.340703 25302 consensus_queue.cc:1035] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:16.349539 25174 consensus_queue.cc:1035] T 630ea99d845540beb727a686457d0e18 P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 234.413 ms
time per row: 0.117207 ms
I20240503 14:12:16.696219 23540 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33990:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:16.705950 23660 tablet_service.cc:1467] Processing CreateTablet for tablet afec3382bc9043dda117d0c7c99c4816 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ba7897d7e9c740fc81086cbb73ed7f3e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.706444 23660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afec3382bc9043dda117d0c7c99c4816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.706490 23923 tablet_service.cc:1467] Processing CreateTablet for tablet afec3382bc9043dda117d0c7c99c4816 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ba7897d7e9c740fc81086cbb73ed7f3e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.706900 23923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afec3382bc9043dda117d0c7c99c4816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.708117 24316 tablet_service.cc:1467] Processing CreateTablet for tablet 01f188dac6914137ba9607c3d7c468cb (DEFAULT_TABLE table=rebalance_test_table_10 [id=ba7897d7e9c740fc81086cbb73ed7f3e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.708585 24316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f188dac6914137ba9607c3d7c468cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.712813 23924 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5addacc5e44689afcc942e217c0ec2 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ba7897d7e9c740fc81086cbb73ed7f3e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.713099 23792 tablet_service.cc:1467] Processing CreateTablet for tablet 01f188dac6914137ba9607c3d7c468cb (DEFAULT_TABLE table=rebalance_test_table_10 [id=ba7897d7e9c740fc81086cbb73ed7f3e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.713224 23924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5addacc5e44689afcc942e217c0ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.713260 24186 tablet_service.cc:1467] Processing CreateTablet for tablet afec3382bc9043dda117d0c7c99c4816 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ba7897d7e9c740fc81086cbb73ed7f3e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.713474 23792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f188dac6914137ba9607c3d7c468cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.713656 24186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afec3382bc9043dda117d0c7c99c4816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.716755 24054 tablet_service.cc:1467] Processing CreateTablet for tablet 01f188dac6914137ba9607c3d7c468cb (DEFAULT_TABLE table=rebalance_test_table_10 [id=ba7897d7e9c740fc81086cbb73ed7f3e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.717186 24054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f188dac6914137ba9607c3d7c468cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.721002 23661 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5addacc5e44689afcc942e217c0ec2 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ba7897d7e9c740fc81086cbb73ed7f3e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.721442 23661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5addacc5e44689afcc942e217c0ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.725701 24185 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5addacc5e44689afcc942e217c0ec2 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ba7897d7e9c740fc81086cbb73ed7f3e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.726112 24185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5addacc5e44689afcc942e217c0ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.731441 25374 tablet_bootstrap.cc:492] T 01f188dac6914137ba9607c3d7c468cb P d4eea540f3c241c1ba6290ca53769163: Bootstrap starting.
I20240503 14:12:16.733686 25374 tablet_bootstrap.cc:654] T 01f188dac6914137ba9607c3d7c468cb P d4eea540f3c241c1ba6290ca53769163: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.740301 25374 tablet_bootstrap.cc:492] T 01f188dac6914137ba9607c3d7c468cb P d4eea540f3c241c1ba6290ca53769163: No bootstrap required, opened a new log
I20240503 14:12:16.740562 25374 ts_tablet_manager.cc:1367] T 01f188dac6914137ba9607c3d7c468cb P d4eea540f3c241c1ba6290ca53769163: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240503 14:12:16.741611 25374 raft_consensus.cc:348] T 01f188dac6914137ba9607c3d7c468cb P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:16.742020 25374 raft_consensus.cc:374] T 01f188dac6914137ba9607c3d7c468cb P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.742170 25374 raft_consensus.cc:729] T 01f188dac6914137ba9607c3d7c468cb P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.742542 25374 consensus_queue.cc:260] T 01f188dac6914137ba9607c3d7c468cb P d4eea540f3c241c1ba6290ca53769163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:16.743427 25374 ts_tablet_manager.cc:1398] T 01f188dac6914137ba9607c3d7c468cb P d4eea540f3c241c1ba6290ca53769163: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:16.744086 25375 tablet_bootstrap.cc:492] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b: Bootstrap starting.
I20240503 14:12:16.750720 25375 tablet_bootstrap.cc:654] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.754020 25378 tablet_bootstrap.cc:492] T 6a5addacc5e44689afcc942e217c0ec2 P 8795a3c2a7014baf9126dea0a6ce1bb4: Bootstrap starting.
I20240503 14:12:16.758648 25373 tablet_bootstrap.cc:492] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74: Bootstrap starting.
I20240503 14:12:16.759020 25378 tablet_bootstrap.cc:654] T 6a5addacc5e44689afcc942e217c0ec2 P 8795a3c2a7014baf9126dea0a6ce1bb4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.760830 25373 tablet_bootstrap.cc:654] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.761305 25377 tablet_bootstrap.cc:492] T 01f188dac6914137ba9607c3d7c468cb P 1e79cc5c647f476c98101f9e48b22b9e: Bootstrap starting.
I20240503 14:12:16.761824 25379 tablet_bootstrap.cc:492] T 6a5addacc5e44689afcc942e217c0ec2 P c9a573f6faf64131b114af1e33c1b308: Bootstrap starting.
I20240503 14:12:16.764235 25379 tablet_bootstrap.cc:654] T 6a5addacc5e44689afcc942e217c0ec2 P c9a573f6faf64131b114af1e33c1b308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.767025 25375 tablet_bootstrap.cc:492] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b: No bootstrap required, opened a new log
I20240503 14:12:16.767192 25377 tablet_bootstrap.cc:654] T 01f188dac6914137ba9607c3d7c468cb P 1e79cc5c647f476c98101f9e48b22b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.767254 25375 ts_tablet_manager.cc:1367] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.005s
I20240503 14:12:16.768122 25375 raft_consensus.cc:348] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:16.768460 25375 raft_consensus.cc:374] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.768602 25375 raft_consensus.cc:729] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.768903 25375 consensus_queue.cc:260] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:16.770468 25375 ts_tablet_manager.cc:1398] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:16.770907 25375 tablet_bootstrap.cc:492] T afec3382bc9043dda117d0c7c99c4816 P d40aeb06cfcc4c3a897b40290d12a15b: Bootstrap starting.
I20240503 14:12:16.776052 25375 tablet_bootstrap.cc:654] T afec3382bc9043dda117d0c7c99c4816 P d40aeb06cfcc4c3a897b40290d12a15b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.784106 25373 tablet_bootstrap.cc:492] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74: No bootstrap required, opened a new log
I20240503 14:12:16.784358 25373 ts_tablet_manager.cc:1367] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.000s
I20240503 14:12:16.785293 25373 raft_consensus.cc:348] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:16.785509 25373 raft_consensus.cc:374] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.785873 25373 raft_consensus.cc:729] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.786258 25373 consensus_queue.cc:260] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:16.787150 25373 ts_tablet_manager.cc:1398] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:16.790288 25375 tablet_bootstrap.cc:492] T afec3382bc9043dda117d0c7c99c4816 P d40aeb06cfcc4c3a897b40290d12a15b: No bootstrap required, opened a new log
I20240503 14:12:16.790515 25375 ts_tablet_manager.cc:1367] T afec3382bc9043dda117d0c7c99c4816 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.003s
I20240503 14:12:16.791354 25375 raft_consensus.cc:348] T afec3382bc9043dda117d0c7c99c4816 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:16.791633 25375 raft_consensus.cc:374] T afec3382bc9043dda117d0c7c99c4816 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.792107 25375 raft_consensus.cc:729] T afec3382bc9043dda117d0c7c99c4816 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.792479 25375 consensus_queue.cc:260] T afec3382bc9043dda117d0c7c99c4816 P d40aeb06cfcc4c3a897b40290d12a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:16.793828 25375 ts_tablet_manager.cc:1398] T afec3382bc9043dda117d0c7c99c4816 P d40aeb06cfcc4c3a897b40290d12a15b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:16.794713 25378 tablet_bootstrap.cc:492] T 6a5addacc5e44689afcc942e217c0ec2 P 8795a3c2a7014baf9126dea0a6ce1bb4: No bootstrap required, opened a new log
I20240503 14:12:16.794880 25378 ts_tablet_manager.cc:1367] T 6a5addacc5e44689afcc942e217c0ec2 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.003s
I20240503 14:12:16.795859 25378 raft_consensus.cc:348] T 6a5addacc5e44689afcc942e217c0ec2 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:16.796207 25378 raft_consensus.cc:374] T 6a5addacc5e44689afcc942e217c0ec2 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.796351 25378 raft_consensus.cc:729] T 6a5addacc5e44689afcc942e217c0ec2 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.796773 25379 tablet_bootstrap.cc:492] T 6a5addacc5e44689afcc942e217c0ec2 P c9a573f6faf64131b114af1e33c1b308: No bootstrap required, opened a new log
I20240503 14:12:16.796667 25378 consensus_queue.cc:260] T 6a5addacc5e44689afcc942e217c0ec2 P 8795a3c2a7014baf9126dea0a6ce1bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:16.796978 25379 ts_tablet_manager.cc:1367] T 6a5addacc5e44689afcc942e217c0ec2 P c9a573f6faf64131b114af1e33c1b308: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.010s
I20240503 14:12:16.797493 25378 ts_tablet_manager.cc:1398] T 6a5addacc5e44689afcc942e217c0ec2 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:16.797843 25378 tablet_bootstrap.cc:492] T afec3382bc9043dda117d0c7c99c4816 P 8795a3c2a7014baf9126dea0a6ce1bb4: Bootstrap starting.
I20240503 14:12:16.798676 25379 raft_consensus.cc:348] T 6a5addacc5e44689afcc942e217c0ec2 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:16.798894 25379 raft_consensus.cc:374] T 6a5addacc5e44689afcc942e217c0ec2 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.799015 25379 raft_consensus.cc:729] T 6a5addacc5e44689afcc942e217c0ec2 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.799280 25379 consensus_queue.cc:260] T 6a5addacc5e44689afcc942e217c0ec2 P c9a573f6faf64131b114af1e33c1b308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:16.800895 25379 ts_tablet_manager.cc:1398] T 6a5addacc5e44689afcc942e217c0ec2 P c9a573f6faf64131b114af1e33c1b308: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:16.801344 25379 tablet_bootstrap.cc:492] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308: Bootstrap starting.
I20240503 14:12:16.803953 25377 tablet_bootstrap.cc:492] T 01f188dac6914137ba9607c3d7c468cb P 1e79cc5c647f476c98101f9e48b22b9e: No bootstrap required, opened a new log
I20240503 14:12:16.804415 25377 ts_tablet_manager.cc:1367] T 01f188dac6914137ba9607c3d7c468cb P 1e79cc5c647f476c98101f9e48b22b9e: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.007s
I20240503 14:12:16.805771 25378 tablet_bootstrap.cc:654] T afec3382bc9043dda117d0c7c99c4816 P 8795a3c2a7014baf9126dea0a6ce1bb4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.805469 25377 raft_consensus.cc:348] T 01f188dac6914137ba9607c3d7c468cb P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:16.805897 25377 raft_consensus.cc:374] T 01f188dac6914137ba9607c3d7c468cb P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.805994 25377 raft_consensus.cc:729] T 01f188dac6914137ba9607c3d7c468cb P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.806279 25377 consensus_queue.cc:260] T 01f188dac6914137ba9607c3d7c468cb P 1e79cc5c647f476c98101f9e48b22b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:16.808375 25377 ts_tablet_manager.cc:1398] T 01f188dac6914137ba9607c3d7c468cb P 1e79cc5c647f476c98101f9e48b22b9e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:16.808692 25379 tablet_bootstrap.cc:654] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.814030 25378 tablet_bootstrap.cc:492] T afec3382bc9043dda117d0c7c99c4816 P 8795a3c2a7014baf9126dea0a6ce1bb4: No bootstrap required, opened a new log
I20240503 14:12:16.814229 25378 ts_tablet_manager.cc:1367] T afec3382bc9043dda117d0c7c99c4816 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20240503 14:12:16.814359 25379 tablet_bootstrap.cc:492] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308: No bootstrap required, opened a new log
I20240503 14:12:16.814507 25379 ts_tablet_manager.cc:1367] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20240503 14:12:16.815011 25378 raft_consensus.cc:348] T afec3382bc9043dda117d0c7c99c4816 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:16.815207 25379 raft_consensus.cc:348] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:16.815428 25379 raft_consensus.cc:374] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.815502 25379 raft_consensus.cc:729] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.815547 25378 raft_consensus.cc:374] T afec3382bc9043dda117d0c7c99c4816 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.815631 25378 raft_consensus.cc:729] T afec3382bc9043dda117d0c7c99c4816 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.815829 25379 consensus_queue.cc:260] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:16.815901 25378 consensus_queue.cc:260] T afec3382bc9043dda117d0c7c99c4816 P 8795a3c2a7014baf9126dea0a6ce1bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:16.816555 25379 ts_tablet_manager.cc:1398] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 14:12:16.817436 25378 ts_tablet_manager.cc:1398] T afec3382bc9043dda117d0c7c99c4816 P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:16.835362 25367 raft_consensus.cc:482] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.835533 25367 raft_consensus.cc:504] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:16.836309 25367 leader_election.cc:288] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175), c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475)
I20240503 14:12:16.836789 24205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5addacc5e44689afcc942e217c0ec2" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308" is_pre_election: true
I20240503 14:12:16.837000 24205 raft_consensus.cc:2454] T 6a5addacc5e44689afcc942e217c0ec2 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 0.
I20240503 14:12:16.837159 23681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5addacc5e44689afcc942e217c0ec2" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" is_pre_election: true
I20240503 14:12:16.837399 23681 raft_consensus.cc:2454] T 6a5addacc5e44689afcc942e217c0ec2 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 0.
I20240503 14:12:16.837771 23878 leader_election.cc:302] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:16.838029 25367 raft_consensus.cc:2781] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:16.838146 25367 raft_consensus.cc:482] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:16.838227 25367 raft_consensus.cc:3037] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.840510 25367 raft_consensus.cc:504] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:16.841212 25367 leader_election.cc:288] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 election: Requested vote from peers 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175), c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475)
I20240503 14:12:16.841624 23681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5addacc5e44689afcc942e217c0ec2" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4"
I20240503 14:12:16.841624 24205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5addacc5e44689afcc942e217c0ec2" candidate_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308"
I20240503 14:12:16.841825 23681 raft_consensus.cc:3037] T 6a5addacc5e44689afcc942e217c0ec2 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.841825 24205 raft_consensus.cc:3037] T 6a5addacc5e44689afcc942e217c0ec2 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.844661 23681 raft_consensus.cc:2454] T 6a5addacc5e44689afcc942e217c0ec2 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 1.
I20240503 14:12:16.844650 24205 raft_consensus.cc:2454] T 6a5addacc5e44689afcc942e217c0ec2 P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d40aeb06cfcc4c3a897b40290d12a15b in term 1.
I20240503 14:12:16.844995 23878 leader_election.cc:302] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:16.845237 25367 raft_consensus.cc:2781] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:16.845391 25367 raft_consensus.cc:686] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [term 1 LEADER]: Becoming Leader. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Running, Role: LEADER
I20240503 14:12:16.845667 25367 consensus_queue.cc:237] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } }
I20240503 14:12:16.848659 23540 catalog_manager.cc:5484] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b reported cstate change: term changed from 0 to 1, leader changed from <none> to d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67). New cstate: current_term: 1 leader_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:16.903460 25261 raft_consensus.cc:482] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.903618 25261 raft_consensus.cc:504] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:16.904353 25261 leader_election.cc:288] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175), d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697)
I20240503 14:12:16.904718 23681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afec3382bc9043dda117d0c7c99c4816" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" is_pre_election: true
I20240503 14:12:16.904837 23943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afec3382bc9043dda117d0c7c99c4816" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" is_pre_election: true
I20240503 14:12:16.904969 23681 raft_consensus.cc:2454] T afec3382bc9043dda117d0c7c99c4816 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 0.
I20240503 14:12:16.905076 23943 raft_consensus.cc:2454] T afec3382bc9043dda117d0c7c99c4816 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 0.
I20240503 14:12:16.905355 24140 leader_election.cc:302] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8795a3c2a7014baf9126dea0a6ce1bb4, c9a573f6faf64131b114af1e33c1b308; no voters:
I20240503 14:12:16.905602 25261 raft_consensus.cc:2781] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:16.905721 25261 raft_consensus.cc:482] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:16.905823 25261 raft_consensus.cc:3037] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.907615 25261 raft_consensus.cc:504] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:16.908190 25261 leader_election.cc:288] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 election: Requested vote from peers 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175), d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697)
I20240503 14:12:16.908561 23681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afec3382bc9043dda117d0c7c99c4816" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4"
I20240503 14:12:16.908659 23943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afec3382bc9043dda117d0c7c99c4816" candidate_uuid: "c9a573f6faf64131b114af1e33c1b308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b"
I20240503 14:12:16.908787 23681 raft_consensus.cc:3037] T afec3382bc9043dda117d0c7c99c4816 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.908851 23943 raft_consensus.cc:3037] T afec3382bc9043dda117d0c7c99c4816 P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.911603 23681 raft_consensus.cc:2454] T afec3382bc9043dda117d0c7c99c4816 P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 1.
I20240503 14:12:16.911617 23943 raft_consensus.cc:2454] T afec3382bc9043dda117d0c7c99c4816 P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a573f6faf64131b114af1e33c1b308 in term 1.
I20240503 14:12:16.912012 24139 leader_election.cc:302] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a573f6faf64131b114af1e33c1b308, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:16.912366 25261 raft_consensus.cc:2781] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:16.912688 25261 raft_consensus.cc:686] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [term 1 LEADER]: Becoming Leader. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Running, Role: LEADER
I20240503 14:12:16.912982 25261 consensus_queue.cc:237] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } }
I20240503 14:12:16.915938 23540 catalog_manager.cc:5484] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9a573f6faf64131b114af1e33c1b308 (127.22.242.69). New cstate: current_term: 1 leader_uuid: "c9a573f6faf64131b114af1e33c1b308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:16.923069 25225 raft_consensus.cc:482] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.923202 25225 raft_consensus.cc:504] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:16.923728 25225 leader_election.cc:288] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675), d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473)
I20240503 14:12:16.924206 24073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f188dac6914137ba9607c3d7c468cb" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e" is_pre_election: true
I20240503 14:12:16.924226 23813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f188dac6914137ba9607c3d7c468cb" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163" is_pre_election: true
I20240503 14:12:16.924392 24073 raft_consensus.cc:2454] T 01f188dac6914137ba9607c3d7c468cb P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 0.
I20240503 14:12:16.924419 23813 raft_consensus.cc:2454] T 01f188dac6914137ba9607c3d7c468cb P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 0.
I20240503 14:12:16.924824 24273 leader_election.cc:302] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:16.925086 25225 raft_consensus.cc:2781] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:16.925194 25225 raft_consensus.cc:482] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:16.925282 25225 raft_consensus.cc:3037] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.927153 25225 raft_consensus.cc:504] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:16.927721 25225 leader_election.cc:288] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 election: Requested vote from peers 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675), d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473)
I20240503 14:12:16.928035 24073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f188dac6914137ba9607c3d7c468cb" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e"
I20240503 14:12:16.928242 24073 raft_consensus.cc:3037] T 01f188dac6914137ba9607c3d7c468cb P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.928144 23813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f188dac6914137ba9607c3d7c468cb" candidate_uuid: "ff53511345114c4ab97a9a8decef3a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163"
I20240503 14:12:16.928365 23813 raft_consensus.cc:3037] T 01f188dac6914137ba9607c3d7c468cb P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.930035 24073 raft_consensus.cc:2454] T 01f188dac6914137ba9607c3d7c468cb P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 1.
I20240503 14:12:16.930188 23813 raft_consensus.cc:2454] T 01f188dac6914137ba9607c3d7c468cb P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff53511345114c4ab97a9a8decef3a74 in term 1.
I20240503 14:12:16.930680 24273 leader_election.cc:302] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:16.930953 25225 raft_consensus.cc:2781] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:16.931075 25225 raft_consensus.cc:686] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [term 1 LEADER]: Becoming Leader. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Running, Role: LEADER
I20240503 14:12:16.931381 25225 consensus_queue.cc:237] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:16.934347 23540 catalog_manager.cc:5484] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff53511345114c4ab97a9a8decef3a74 (127.22.242.70). New cstate: current_term: 1 leader_uuid: "ff53511345114c4ab97a9a8decef3a74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:17.329188 25217 consensus_queue.cc:1035] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:17.334515 25392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:17.335045 25392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:12:17.336194 25217 consensus_queue.cc:1035] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:17.347527 25392 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240503 14:12:17.398931 25225 consensus_queue.cc:1035] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:17.409801 25225 consensus_queue.cc:1035] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:17.425559 25261 consensus_queue.cc:1035] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:17.443457 25261 consensus_queue.cc:1035] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 278.423 ms
time per row: 0.139212 ms
I20240503 14:12:17.794832 23532 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33990:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:17.804085 24054 tablet_service.cc:1467] Processing CreateTablet for tablet c1a414d5fd374961afe51938583bf66d (DEFAULT_TABLE table=rebalance_test_table_11 [id=4203be9fc848431aa33d211ba9c3b25a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.804492 24316 tablet_service.cc:1467] Processing CreateTablet for tablet c1a414d5fd374961afe51938583bf66d (DEFAULT_TABLE table=rebalance_test_table_11 [id=4203be9fc848431aa33d211ba9c3b25a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.804641 24054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a414d5fd374961afe51938583bf66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.804579 23792 tablet_service.cc:1467] Processing CreateTablet for tablet c1a414d5fd374961afe51938583bf66d (DEFAULT_TABLE table=rebalance_test_table_11 [id=4203be9fc848431aa33d211ba9c3b25a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.804899 24316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a414d5fd374961afe51938583bf66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.805011 23792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a414d5fd374961afe51938583bf66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.806121 23793 tablet_service.cc:1467] Processing CreateTablet for tablet f61c1042cd3042e4b85bc95a83c7f343 (DEFAULT_TABLE table=rebalance_test_table_11 [id=4203be9fc848431aa33d211ba9c3b25a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.806488 23793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61c1042cd3042e4b85bc95a83c7f343. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.811661 23924 tablet_service.cc:1467] Processing CreateTablet for tablet eb9d169944be45e7b73a0f842de8be1e (DEFAULT_TABLE table=rebalance_test_table_11 [id=4203be9fc848431aa33d211ba9c3b25a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.811908 24185 tablet_service.cc:1467] Processing CreateTablet for tablet eb9d169944be45e7b73a0f842de8be1e (DEFAULT_TABLE table=rebalance_test_table_11 [id=4203be9fc848431aa33d211ba9c3b25a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.812096 23924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9d169944be45e7b73a0f842de8be1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.812431 24185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9d169944be45e7b73a0f842de8be1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.812419 24317 tablet_service.cc:1467] Processing CreateTablet for tablet f61c1042cd3042e4b85bc95a83c7f343 (DEFAULT_TABLE table=rebalance_test_table_11 [id=4203be9fc848431aa33d211ba9c3b25a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.812790 24317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61c1042cd3042e4b85bc95a83c7f343. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.815945 25456 tablet_bootstrap.cc:492] T c1a414d5fd374961afe51938583bf66d P ff53511345114c4ab97a9a8decef3a74: Bootstrap starting.
I20240503 14:12:17.816584 23661 tablet_service.cc:1467] Processing CreateTablet for tablet eb9d169944be45e7b73a0f842de8be1e (DEFAULT_TABLE table=rebalance_test_table_11 [id=4203be9fc848431aa33d211ba9c3b25a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.817032 23661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9d169944be45e7b73a0f842de8be1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.819005 25456 tablet_bootstrap.cc:654] T c1a414d5fd374961afe51938583bf66d P ff53511345114c4ab97a9a8decef3a74: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.820633 25457 tablet_bootstrap.cc:492] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e: Bootstrap starting.
I20240503 14:12:17.821161 24055 tablet_service.cc:1467] Processing CreateTablet for tablet f61c1042cd3042e4b85bc95a83c7f343 (DEFAULT_TABLE table=rebalance_test_table_11 [id=4203be9fc848431aa33d211ba9c3b25a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.821592 24055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61c1042cd3042e4b85bc95a83c7f343. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.823527 25457 tablet_bootstrap.cc:654] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.826929 25460 tablet_bootstrap.cc:492] T c1a414d5fd374961afe51938583bf66d P d4eea540f3c241c1ba6290ca53769163: Bootstrap starting.
I20240503 14:12:17.828941 25459 tablet_bootstrap.cc:492] T eb9d169944be45e7b73a0f842de8be1e P d40aeb06cfcc4c3a897b40290d12a15b: Bootstrap starting.
I20240503 14:12:17.829224 25460 tablet_bootstrap.cc:654] T c1a414d5fd374961afe51938583bf66d P d4eea540f3c241c1ba6290ca53769163: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.831251 25459 tablet_bootstrap.cc:654] T eb9d169944be45e7b73a0f842de8be1e P d40aeb06cfcc4c3a897b40290d12a15b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.832662 25456 tablet_bootstrap.cc:492] T c1a414d5fd374961afe51938583bf66d P ff53511345114c4ab97a9a8decef3a74: No bootstrap required, opened a new log
I20240503 14:12:17.832820 25456 ts_tablet_manager.cc:1367] T c1a414d5fd374961afe51938583bf66d P ff53511345114c4ab97a9a8decef3a74: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
I20240503 14:12:17.833580 25456 raft_consensus.cc:348] T c1a414d5fd374961afe51938583bf66d P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:17.833827 25456 raft_consensus.cc:374] T c1a414d5fd374961afe51938583bf66d P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.833927 25456 raft_consensus.cc:729] T c1a414d5fd374961afe51938583bf66d P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.834210 25456 consensus_queue.cc:260] T c1a414d5fd374961afe51938583bf66d P ff53511345114c4ab97a9a8decef3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:17.836035 25456 ts_tablet_manager.cc:1398] T c1a414d5fd374961afe51938583bf66d P ff53511345114c4ab97a9a8decef3a74: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:17.836328 25456 tablet_bootstrap.cc:492] T f61c1042cd3042e4b85bc95a83c7f343 P ff53511345114c4ab97a9a8decef3a74: Bootstrap starting.
I20240503 14:12:17.839639 25457 tablet_bootstrap.cc:492] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e: No bootstrap required, opened a new log
I20240503 14:12:17.839965 25457 ts_tablet_manager.cc:1367] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.006s
I20240503 14:12:17.842295 25456 tablet_bootstrap.cc:654] T f61c1042cd3042e4b85bc95a83c7f343 P ff53511345114c4ab97a9a8decef3a74: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.843617 25459 tablet_bootstrap.cc:492] T eb9d169944be45e7b73a0f842de8be1e P d40aeb06cfcc4c3a897b40290d12a15b: No bootstrap required, opened a new log
I20240503 14:12:17.843832 25459 ts_tablet_manager.cc:1367] T eb9d169944be45e7b73a0f842de8be1e P d40aeb06cfcc4c3a897b40290d12a15b: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20240503 14:12:17.841984 25457 raft_consensus.cc:348] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:17.844878 25457 raft_consensus.cc:374] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.845034 25457 raft_consensus.cc:729] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.844959 25459 raft_consensus.cc:348] T eb9d169944be45e7b73a0f842de8be1e P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:17.845378 25459 raft_consensus.cc:374] T eb9d169944be45e7b73a0f842de8be1e P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.845335 25457 consensus_queue.cc:260] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:17.845525 25459 raft_consensus.cc:729] T eb9d169944be45e7b73a0f842de8be1e P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40aeb06cfcc4c3a897b40290d12a15b, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.846089 25457 ts_tablet_manager.cc:1398] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:17.846374 25457 tablet_bootstrap.cc:492] T f61c1042cd3042e4b85bc95a83c7f343 P 1e79cc5c647f476c98101f9e48b22b9e: Bootstrap starting.
I20240503 14:12:17.848256 25457 tablet_bootstrap.cc:654] T f61c1042cd3042e4b85bc95a83c7f343 P 1e79cc5c647f476c98101f9e48b22b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.853296 25459 consensus_queue.cc:260] T eb9d169944be45e7b73a0f842de8be1e P d40aeb06cfcc4c3a897b40290d12a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:17.854559 25459 ts_tablet_manager.cc:1398] T eb9d169944be45e7b73a0f842de8be1e P d40aeb06cfcc4c3a897b40290d12a15b: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240503 14:12:17.855651 25466 tablet_bootstrap.cc:492] T eb9d169944be45e7b73a0f842de8be1e P c9a573f6faf64131b114af1e33c1b308: Bootstrap starting.
I20240503 14:12:17.856340 25465 tablet_bootstrap.cc:492] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4: Bootstrap starting.
I20240503 14:12:17.860520 25465 tablet_bootstrap.cc:654] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.860837 25457 tablet_bootstrap.cc:492] T f61c1042cd3042e4b85bc95a83c7f343 P 1e79cc5c647f476c98101f9e48b22b9e: No bootstrap required, opened a new log
I20240503 14:12:17.861006 25457 ts_tablet_manager.cc:1367] T f61c1042cd3042e4b85bc95a83c7f343 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20240503 14:12:17.861593 25457 raft_consensus.cc:348] T f61c1042cd3042e4b85bc95a83c7f343 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:17.861770 25466 tablet_bootstrap.cc:654] T eb9d169944be45e7b73a0f842de8be1e P c9a573f6faf64131b114af1e33c1b308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.861884 25457 raft_consensus.cc:374] T f61c1042cd3042e4b85bc95a83c7f343 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.861979 25457 raft_consensus.cc:729] T f61c1042cd3042e4b85bc95a83c7f343 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.862223 25457 consensus_queue.cc:260] T f61c1042cd3042e4b85bc95a83c7f343 P 1e79cc5c647f476c98101f9e48b22b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:17.864558 25457 ts_tablet_manager.cc:1398] T f61c1042cd3042e4b85bc95a83c7f343 P 1e79cc5c647f476c98101f9e48b22b9e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:17.866345 25466 tablet_bootstrap.cc:492] T eb9d169944be45e7b73a0f842de8be1e P c9a573f6faf64131b114af1e33c1b308: No bootstrap required, opened a new log
I20240503 14:12:17.866443 25456 tablet_bootstrap.cc:492] T f61c1042cd3042e4b85bc95a83c7f343 P ff53511345114c4ab97a9a8decef3a74: No bootstrap required, opened a new log
I20240503 14:12:17.866616 25466 ts_tablet_manager.cc:1367] T eb9d169944be45e7b73a0f842de8be1e P c9a573f6faf64131b114af1e33c1b308: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 14:12:17.866636 25456 ts_tablet_manager.cc:1367] T f61c1042cd3042e4b85bc95a83c7f343 P ff53511345114c4ab97a9a8decef3a74: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.004s
I20240503 14:12:17.867470 25466 raft_consensus.cc:348] T eb9d169944be45e7b73a0f842de8be1e P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:17.867430 25456 raft_consensus.cc:348] T f61c1042cd3042e4b85bc95a83c7f343 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:17.867689 25456 raft_consensus.cc:374] T f61c1042cd3042e4b85bc95a83c7f343 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.867769 25456 raft_consensus.cc:729] T f61c1042cd3042e4b85bc95a83c7f343 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff53511345114c4ab97a9a8decef3a74, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.867888 25466 raft_consensus.cc:374] T eb9d169944be45e7b73a0f842de8be1e P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.868023 25466 raft_consensus.cc:729] T eb9d169944be45e7b73a0f842de8be1e P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a573f6faf64131b114af1e33c1b308, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.868101 25456 consensus_queue.cc:260] T f61c1042cd3042e4b85bc95a83c7f343 P ff53511345114c4ab97a9a8decef3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:17.868407 25466 consensus_queue.cc:260] T eb9d169944be45e7b73a0f842de8be1e P c9a573f6faf64131b114af1e33c1b308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:17.869511 25456 ts_tablet_manager.cc:1398] T f61c1042cd3042e4b85bc95a83c7f343 P ff53511345114c4ab97a9a8decef3a74: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:17.869920 25466 ts_tablet_manager.cc:1398] T eb9d169944be45e7b73a0f842de8be1e P c9a573f6faf64131b114af1e33c1b308: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:17.870718 25465 tablet_bootstrap.cc:492] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4: No bootstrap required, opened a new log
I20240503 14:12:17.870872 25465 ts_tablet_manager.cc:1367] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20240503 14:12:17.870909 25460 tablet_bootstrap.cc:492] T c1a414d5fd374961afe51938583bf66d P d4eea540f3c241c1ba6290ca53769163: No bootstrap required, opened a new log
I20240503 14:12:17.871121 25460 ts_tablet_manager.cc:1367] T c1a414d5fd374961afe51938583bf66d P d4eea540f3c241c1ba6290ca53769163: Time spent bootstrapping tablet: real 0.044s user 0.016s sys 0.012s
I20240503 14:12:17.871723 25465 raft_consensus.cc:348] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:17.871948 25465 raft_consensus.cc:374] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.872057 25465 raft_consensus.cc:729] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.871982 25460 raft_consensus.cc:348] T c1a414d5fd374961afe51938583bf66d P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:17.872274 25460 raft_consensus.cc:374] T c1a414d5fd374961afe51938583bf66d P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.872385 25460 raft_consensus.cc:729] T c1a414d5fd374961afe51938583bf66d P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.872340 25465 consensus_queue.cc:260] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:17.872630 25460 consensus_queue.cc:260] T c1a414d5fd374961afe51938583bf66d P d4eea540f3c241c1ba6290ca53769163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:17.873184 25460 ts_tablet_manager.cc:1398] T c1a414d5fd374961afe51938583bf66d P d4eea540f3c241c1ba6290ca53769163: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:17.873477 25460 tablet_bootstrap.cc:492] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163: Bootstrap starting.
I20240503 14:12:17.873636 25465 ts_tablet_manager.cc:1398] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:17.875136 25460 tablet_bootstrap.cc:654] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.879460 25460 tablet_bootstrap.cc:492] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163: No bootstrap required, opened a new log
I20240503 14:12:17.879681 25460 ts_tablet_manager.cc:1367] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240503 14:12:17.880589 25460 raft_consensus.cc:348] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:17.880801 25460 raft_consensus.cc:374] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.880887 25460 raft_consensus.cc:729] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.881155 25460 consensus_queue.cc:260] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:17.881722 25460 ts_tablet_manager.cc:1398] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:17.936985 25356 raft_consensus.cc:482] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.937192 25356 raft_consensus.cc:504] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:17.937981 25356 leader_election.cc:288] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067), 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675)
I20240503 14:12:17.938372 24336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61c1042cd3042e4b85bc95a83c7f343" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74" is_pre_election: true
I20240503 14:12:17.938463 24073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61c1042cd3042e4b85bc95a83c7f343" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e" is_pre_election: true
I20240503 14:12:17.938591 24336 raft_consensus.cc:2454] T f61c1042cd3042e4b85bc95a83c7f343 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 0.
I20240503 14:12:17.938720 24073 raft_consensus.cc:2454] T f61c1042cd3042e4b85bc95a83c7f343 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 0.
I20240503 14:12:17.939060 23747 leader_election.cc:302] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:17.939327 25356 raft_consensus.cc:2781] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:17.939424 25356 raft_consensus.cc:482] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:17.939510 25356 raft_consensus.cc:3037] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.941458 25356 raft_consensus.cc:504] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:17.941984 25356 leader_election.cc:288] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 election: Requested vote from peers ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067), 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675)
I20240503 14:12:17.942481 24336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61c1042cd3042e4b85bc95a83c7f343" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74"
I20240503 14:12:17.942481 24073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61c1042cd3042e4b85bc95a83c7f343" candidate_uuid: "d4eea540f3c241c1ba6290ca53769163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e79cc5c647f476c98101f9e48b22b9e"
I20240503 14:12:17.942713 24336 raft_consensus.cc:3037] T f61c1042cd3042e4b85bc95a83c7f343 P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.942713 24073 raft_consensus.cc:3037] T f61c1042cd3042e4b85bc95a83c7f343 P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.945551 24336 raft_consensus.cc:2454] T f61c1042cd3042e4b85bc95a83c7f343 P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 1.
I20240503 14:12:17.945551 24073 raft_consensus.cc:2454] T f61c1042cd3042e4b85bc95a83c7f343 P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4eea540f3c241c1ba6290ca53769163 in term 1.
I20240503 14:12:17.946067 23747 leader_election.cc:302] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4eea540f3c241c1ba6290ca53769163, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:17.946420 25356 raft_consensus.cc:2781] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:17.946763 25356 raft_consensus.cc:686] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [term 1 LEADER]: Becoming Leader. State: Replica: d4eea540f3c241c1ba6290ca53769163, State: Running, Role: LEADER
I20240503 14:12:17.947024 25356 consensus_queue.cc:237] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:17.950160 23540 catalog_manager.cc:5484] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4eea540f3c241c1ba6290ca53769163 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "d4eea540f3c241c1ba6290ca53769163" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:17.998805 25301 raft_consensus.cc:482] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.998975 25301 raft_consensus.cc:504] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:17.999796 25301 leader_election.cc:288] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475), d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697)
I20240503 14:12:18.000268 24205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9d169944be45e7b73a0f842de8be1e" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308" is_pre_election: true
I20240503 14:12:18.000296 23943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9d169944be45e7b73a0f842de8be1e" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" is_pre_election: true
I20240503 14:12:18.000492 24205 raft_consensus.cc:2454] T eb9d169944be45e7b73a0f842de8be1e P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 0.
I20240503 14:12:18.000546 23943 raft_consensus.cc:2454] T eb9d169944be45e7b73a0f842de8be1e P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 0.
I20240503 14:12:18.000982 23614 leader_election.cc:302] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8795a3c2a7014baf9126dea0a6ce1bb4, d40aeb06cfcc4c3a897b40290d12a15b; no voters:
I20240503 14:12:18.001232 25473 raft_consensus.cc:2781] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:18.001322 25473 raft_consensus.cc:482] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:18.001446 25473 raft_consensus.cc:3037] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.003304 25473 raft_consensus.cc:504] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:18.003892 25473 leader_election.cc:288] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 election: Requested vote from peers c9a573f6faf64131b114af1e33c1b308 (127.22.242.69:45475), d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697)
I20240503 14:12:18.004371 23943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9d169944be45e7b73a0f842de8be1e" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40aeb06cfcc4c3a897b40290d12a15b"
I20240503 14:12:18.004426 24205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9d169944be45e7b73a0f842de8be1e" candidate_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a573f6faf64131b114af1e33c1b308"
I20240503 14:12:18.004603 24205 raft_consensus.cc:3037] T eb9d169944be45e7b73a0f842de8be1e P c9a573f6faf64131b114af1e33c1b308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.004601 23943 raft_consensus.cc:3037] T eb9d169944be45e7b73a0f842de8be1e P d40aeb06cfcc4c3a897b40290d12a15b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.007321 24205 raft_consensus.cc:2454] T eb9d169944be45e7b73a0f842de8be1e P c9a573f6faf64131b114af1e33c1b308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 1.
I20240503 14:12:18.007323 23943 raft_consensus.cc:2454] T eb9d169944be45e7b73a0f842de8be1e P d40aeb06cfcc4c3a897b40290d12a15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8795a3c2a7014baf9126dea0a6ce1bb4 in term 1.
I20240503 14:12:18.007683 23615 leader_election.cc:302] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8795a3c2a7014baf9126dea0a6ce1bb4, c9a573f6faf64131b114af1e33c1b308; no voters:
I20240503 14:12:18.007962 25473 raft_consensus.cc:2781] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:18.008138 25473 raft_consensus.cc:686] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [term 1 LEADER]: Becoming Leader. State: Replica: 8795a3c2a7014baf9126dea0a6ce1bb4, State: Running, Role: LEADER
I20240503 14:12:18.008411 25473 consensus_queue.cc:237] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } }
I20240503 14:12:18.011384 23540 catalog_manager.cc:5484] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8795a3c2a7014baf9126dea0a6ce1bb4" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 44175 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:18.048851 25080 raft_consensus.cc:482] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.049028 25080 raft_consensus.cc:504] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:18.049731 25080 leader_election.cc:288] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067), d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473)
I20240503 14:12:18.050225 24336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a414d5fd374961afe51938583bf66d" candidate_uuid: "1e79cc5c647f476c98101f9e48b22b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74" is_pre_election: true
I20240503 14:12:18.050451 24336 raft_consensus.cc:2454] T c1a414d5fd374961afe51938583bf66d P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e79cc5c647f476c98101f9e48b22b9e in term 0.
I20240503 14:12:18.050451 23813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a414d5fd374961afe51938583bf66d" candidate_uuid: "1e79cc5c647f476c98101f9e48b22b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163" is_pre_election: true
I20240503 14:12:18.050700 23813 raft_consensus.cc:2454] T c1a414d5fd374961afe51938583bf66d P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e79cc5c647f476c98101f9e48b22b9e in term 0.
I20240503 14:12:18.050926 24009 leader_election.cc:302] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:18.051198 25080 raft_consensus.cc:2781] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:18.051329 25080 raft_consensus.cc:482] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:18.051415 25080 raft_consensus.cc:3037] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.053471 25080 raft_consensus.cc:504] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:18.054019 25080 leader_election.cc:288] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [CANDIDATE]: Term 1 election: Requested vote from peers ff53511345114c4ab97a9a8decef3a74 (127.22.242.70:35067), d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473)
I20240503 14:12:18.054371 24336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a414d5fd374961afe51938583bf66d" candidate_uuid: "1e79cc5c647f476c98101f9e48b22b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff53511345114c4ab97a9a8decef3a74"
I20240503 14:12:18.054457 23813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a414d5fd374961afe51938583bf66d" candidate_uuid: "1e79cc5c647f476c98101f9e48b22b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4eea540f3c241c1ba6290ca53769163"
I20240503 14:12:18.054549 24336 raft_consensus.cc:3037] T c1a414d5fd374961afe51938583bf66d P ff53511345114c4ab97a9a8decef3a74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.054677 23813 raft_consensus.cc:3037] T c1a414d5fd374961afe51938583bf66d P d4eea540f3c241c1ba6290ca53769163 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.056481 24336 raft_consensus.cc:2454] T c1a414d5fd374961afe51938583bf66d P ff53511345114c4ab97a9a8decef3a74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e79cc5c647f476c98101f9e48b22b9e in term 1.
I20240503 14:12:18.056589 23813 raft_consensus.cc:2454] T c1a414d5fd374961afe51938583bf66d P d4eea540f3c241c1ba6290ca53769163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e79cc5c647f476c98101f9e48b22b9e in term 1.
I20240503 14:12:18.056939 24009 leader_election.cc:302] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e79cc5c647f476c98101f9e48b22b9e, ff53511345114c4ab97a9a8decef3a74; no voters:
I20240503 14:12:18.057225 25080 raft_consensus.cc:2781] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:18.057598 25080 raft_consensus.cc:686] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [term 1 LEADER]: Becoming Leader. State: Replica: 1e79cc5c647f476c98101f9e48b22b9e, State: Running, Role: LEADER
I20240503 14:12:18.057895 25080 consensus_queue.cc:237] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } }
I20240503 14:12:18.060899 23540 catalog_manager.cc:5484] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68). New cstate: current_term: 1 leader_uuid: "1e79cc5c647f476c98101f9e48b22b9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:18.386708 25356 consensus_queue.cc:1035] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:18.394703 25471 consensus_queue.cc:1035] T f61c1042cd3042e4b85bc95a83c7f343 P d4eea540f3c241c1ba6290ca53769163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e79cc5c647f476c98101f9e48b22b9e" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:18.453934 25473 consensus_queue.cc:1035] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d40aeb06cfcc4c3a897b40290d12a15b" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:18.468410 25301 consensus_queue.cc:1035] T eb9d169944be45e7b73a0f842de8be1e P 8795a3c2a7014baf9126dea0a6ce1bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a573f6faf64131b114af1e33c1b308" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 45475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:18.472009 25477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20240503 14:12:18.472268 25080 consensus_queue.cc:1035] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff53511345114c4ab97a9a8decef3a74" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:18.472592 25477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:12:18.487766 25475 consensus_queue.cc:1035] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4eea540f3c241c1ba6290ca53769163" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:18.488634 25477 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
W20240503 14:12:18.738905 24273 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.66:46473, user_credentials={real_user=slave}} blocked reactor thread for 64459.6us
INSERT report
rows total: 2000
time total: 311.058 ms
time per row: 0.155529 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:12:20.506696 24055 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:20.509044 23661 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:20.511534 23924 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:20.531179 24185 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:20.556759 23792 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:12:20.598892 24317 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:20.815363 24055 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:20.834367 23661 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:20.835453 24185 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:20.846539 23792 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:12:20.859292 24317 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:20.863859 23924 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:12:20.900197 23747 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.70:35067, user_credentials={real_user=slave}} blocked reactor thread for 69399.6us
I20240503 14:12:21.147725 24055 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:21.160259 23661 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:21.166074 23924 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:21.168716 24185 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:21.188364 23792 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:12:21.191967 24317 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:21.432920 24055 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:21.442283 24185 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:21.450331 23661 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:21.468400 23924 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:21.480522 24317 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:21.480726 23792 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:12:21.804661 23924 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:21.807238 24055 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:21.812911 23661 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:21.828687 24185 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:21.830412 23792 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:12:21.839869 24317 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:22.222931 24055 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:22.243690 23661 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:22.249969 24185 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:22.269284 24317 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:22.269740 23924 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:22.305030 23792 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
45604633057f43e9ad77fa7396eacc2b | 127.22.242.126:33953 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.242.84:37849 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
1e79cc5c647f476c98101f9e48b22b9e | 127.22.242.68:37675 | HEALTHY | /B | 3 | 0
8795a3c2a7014baf9126dea0a6ce1bb4 | 127.22.242.65:44175 | HEALTHY | /A | 5 | 0
c9a573f6faf64131b114af1e33c1b308 | 127.22.242.69:45475 | HEALTHY | /C | 6 | 0
d40aeb06cfcc4c3a897b40290d12a15b | 127.22.242.67:37697 | HEALTHY | /B | 7 | 0
d4eea540f3c241c1ba6290ca53769163 | 127.22.242.66:46473 | HEALTHY | /A | 9 | 0
ff53511345114c4ab97a9a8decef3a74 | 127.22.242.70:35067 | HEALTHY | /C | 6 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/C | 2
/B | 2
/A | 2
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.22.242.65 | experimental | 127.22.242.65:44175
local_ip_for_outbound_sockets | 127.22.242.66 | experimental | 127.22.242.66:46473
local_ip_for_outbound_sockets | 127.22.242.67 | experimental | 127.22.242.67:37697
local_ip_for_outbound_sockets | 127.22.242.68 | experimental | 127.22.242.68:37675
local_ip_for_outbound_sockets | 127.22.242.69 | experimental | 127.22.242.69:45475
local_ip_for_outbound_sockets | 127.22.242.70 | experimental | 127.22.242.70:35067
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.242.65:44175
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.242.66:46473
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.242.67:37697
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.22.242.68:37675
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.242.69:45475
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.242.70:35067
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.242.84:37849 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_10 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_11 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 18
Median | 18
Third Quartile | 18
Maximum | 18
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 12
Tablets | 36
Replicas | 108
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:12:22.367512 23497 log_verifier.cc:126] Checking tablet 01f188dac6914137ba9607c3d7c468cb
I20240503 14:12:22.406924 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 01f188dac6914137ba9607c3d7c468cb
I20240503 14:12:22.407104 23497 log_verifier.cc:126] Checking tablet 16e304b5783546aea642a1525767007e
I20240503 14:12:22.444646 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 16e304b5783546aea642a1525767007e
I20240503 14:12:22.444777 23497 log_verifier.cc:126] Checking tablet 174f75183c8f44c2a07ef326177d9d71
I20240503 14:12:22.481738 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 174f75183c8f44c2a07ef326177d9d71
I20240503 14:12:22.481850 23497 log_verifier.cc:126] Checking tablet 2cfc5cdabc784d1ea5f74b066f4e2be8
I20240503 14:12:22.522176 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2cfc5cdabc784d1ea5f74b066f4e2be8
I20240503 14:12:22.522384 23497 log_verifier.cc:126] Checking tablet 32781a4a39434a96aa36a9d896a58786
I20240503 14:12:22.564908 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 32781a4a39434a96aa36a9d896a58786
I20240503 14:12:22.565019 23497 log_verifier.cc:126] Checking tablet 3b562d2ac0524913b7eaded7a0441958
I20240503 14:12:22.604506 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3b562d2ac0524913b7eaded7a0441958
I20240503 14:12:22.604615 23497 log_verifier.cc:126] Checking tablet 5c55248037ca497bbc153b5933d4230e
I20240503 14:12:22.646704 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5c55248037ca497bbc153b5933d4230e
I20240503 14:12:22.646823 23497 log_verifier.cc:126] Checking tablet 630ea99d845540beb727a686457d0e18
I20240503 14:12:22.687883 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 630ea99d845540beb727a686457d0e18
I20240503 14:12:22.687995 23497 log_verifier.cc:126] Checking tablet 666732f9b38f4a20a667bbb9880ece02
I20240503 14:12:22.736133 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 666732f9b38f4a20a667bbb9880ece02
I20240503 14:12:22.736286 23497 log_verifier.cc:126] Checking tablet 6684b0c4e392418cb7c9d64e4eb62c01
I20240503 14:12:22.784096 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6684b0c4e392418cb7c9d64e4eb62c01
I20240503 14:12:22.784297 23497 log_verifier.cc:126] Checking tablet 6a5addacc5e44689afcc942e217c0ec2
I20240503 14:12:22.829955 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6a5addacc5e44689afcc942e217c0ec2
I20240503 14:12:22.830070 23497 log_verifier.cc:126] Checking tablet 7159aeda34f945cd8726b2b7e9ccfdff
I20240503 14:12:22.871112 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7159aeda34f945cd8726b2b7e9ccfdff
I20240503 14:12:22.871217 23497 log_verifier.cc:126] Checking tablet 916b8dad2a52497799b76127ce499a2f
I20240503 14:12:22.910652 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 916b8dad2a52497799b76127ce499a2f
I20240503 14:12:22.910771 23497 log_verifier.cc:126] Checking tablet 944a3478fc06494384f0956bf575b8e6
I20240503 14:12:22.951404 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 944a3478fc06494384f0956bf575b8e6
I20240503 14:12:22.951516 23497 log_verifier.cc:126] Checking tablet 9814abfe435b4bdf9fb65d65d03bc1b9
I20240503 14:12:22.992326 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9814abfe435b4bdf9fb65d65d03bc1b9
I20240503 14:12:22.992437 23497 log_verifier.cc:126] Checking tablet 9e6476b88ba546cc82d721f84d6084df
I20240503 14:12:23.033124 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9e6476b88ba546cc82d721f84d6084df
I20240503 14:12:23.033246 23497 log_verifier.cc:126] Checking tablet a78b4e88ed1e47908a4a2beae34a3ab7
I20240503 14:12:23.072242 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a78b4e88ed1e47908a4a2beae34a3ab7
I20240503 14:12:23.072362 23497 log_verifier.cc:126] Checking tablet abf6fde8492d459cb00bc324e8dbc66b
I20240503 14:12:23.113127 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet abf6fde8492d459cb00bc324e8dbc66b
I20240503 14:12:23.113235 23497 log_verifier.cc:126] Checking tablet ac35b944198044e1af026f24d35e0b7b
I20240503 14:12:23.153921 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ac35b944198044e1af026f24d35e0b7b
I20240503 14:12:23.154038 23497 log_verifier.cc:126] Checking tablet afec3382bc9043dda117d0c7c99c4816
I20240503 14:12:23.195751 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet afec3382bc9043dda117d0c7c99c4816
I20240503 14:12:23.195891 23497 log_verifier.cc:126] Checking tablet c1a414d5fd374961afe51938583bf66d
I20240503 14:12:23.237470 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c1a414d5fd374961afe51938583bf66d
I20240503 14:12:23.237583 23497 log_verifier.cc:126] Checking tablet c306f7c2c4d7493990bdf5242238a381
I20240503 14:12:23.278775 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c306f7c2c4d7493990bdf5242238a381
I20240503 14:12:23.278899 23497 log_verifier.cc:126] Checking tablet caf778976e774eabab30aa682d6bd5a0
I20240503 14:12:23.320608 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet caf778976e774eabab30aa682d6bd5a0
I20240503 14:12:23.320746 23497 log_verifier.cc:126] Checking tablet cf6dcac0c61c4c5fa12add8298c29dcf
I20240503 14:12:23.361475 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet cf6dcac0c61c4c5fa12add8298c29dcf
I20240503 14:12:23.361589 23497 log_verifier.cc:126] Checking tablet d31c3fbce3a74c32aff549e9d1ceda13
I20240503 14:12:23.401275 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d31c3fbce3a74c32aff549e9d1ceda13
I20240503 14:12:23.401392 23497 log_verifier.cc:126] Checking tablet d70414f531ac463c9e89e2236d94250c
I20240503 14:12:23.442404 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d70414f531ac463c9e89e2236d94250c
I20240503 14:12:23.442539 23497 log_verifier.cc:126] Checking tablet def4596ad4634265a15ce77a54833853
I20240503 14:12:23.484269 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet def4596ad4634265a15ce77a54833853
I20240503 14:12:23.484378 23497 log_verifier.cc:126] Checking tablet e0a3e437577742b1807b24a193ebc82b
I20240503 14:12:23.526098 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e0a3e437577742b1807b24a193ebc82b
I20240503 14:12:23.526257 23497 log_verifier.cc:126] Checking tablet e99fc624e14a4e17a010e42e6c3960a2
I20240503 14:12:23.568799 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e99fc624e14a4e17a010e42e6c3960a2
I20240503 14:12:23.568913 23497 log_verifier.cc:126] Checking tablet e9c2d15fb0434a22afc3191ed8df3aac
I20240503 14:12:23.609887 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e9c2d15fb0434a22afc3191ed8df3aac
I20240503 14:12:23.610005 23497 log_verifier.cc:126] Checking tablet eb9d169944be45e7b73a0f842de8be1e
I20240503 14:12:23.650943 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet eb9d169944be45e7b73a0f842de8be1e
I20240503 14:12:23.651048 23497 log_verifier.cc:126] Checking tablet efd2ded4212a49d38a35ef6e07650b74
I20240503 14:12:23.691366 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet efd2ded4212a49d38a35ef6e07650b74
I20240503 14:12:23.691505 23497 log_verifier.cc:126] Checking tablet f08228ffd3984f45afd068fab3eb9a43
I20240503 14:12:23.742731 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f08228ffd3984f45afd068fab3eb9a43
I20240503 14:12:23.742858 23497 log_verifier.cc:126] Checking tablet f1c9262f9e154fe5b30c2ec66b26e4d5
I20240503 14:12:23.791908 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f1c9262f9e154fe5b30c2ec66b26e4d5
I20240503 14:12:23.792048 23497 log_verifier.cc:126] Checking tablet f61c1042cd3042e4b85bc95a83c7f343
I20240503 14:12:23.835368 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f61c1042cd3042e4b85bc95a83c7f343
I20240503 14:12:23.835487 23497 log_verifier.cc:126] Checking tablet fa04297e5841498caad1c4bf4b851fe4
I20240503 14:12:23.876688 23497 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fa04297e5841498caad1c4bf4b851fe4
I20240503 14:12:23.896100 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 23599
I20240503 14:12:23.902287 23720 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:24.037359 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 23599
W20240503 14:12:24.062717 24140 connection.cc:535] client connection to 127.22.242.65:44175 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:12:24.062749 23878 connection.cc:535] client connection to 127.22.242.65:44175 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:12:24.062965 23878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:12:24.062983 24140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:12:24.063333 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 23730
W20240503 14:12:24.065387 23878 consensus_peers.cc:485] T 16e304b5783546aea642a1525767007e P d40aeb06cfcc4c3a897b40290d12a15b -> Peer 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175): Couldn't send request to peer 8795a3c2a7014baf9126dea0a6ce1bb4. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:44175: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:24.065657 24140 consensus_peers.cc:485] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 -> Peer 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175): Couldn't send request to peer 8795a3c2a7014baf9126dea0a6ce1bb4. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:44175: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:24.070040 23853 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:12:24.093770 23878 consensus_peers.cc:485] T 6684b0c4e392418cb7c9d64e4eb62c01 P d40aeb06cfcc4c3a897b40290d12a15b -> Peer 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175): Couldn't send request to peer 8795a3c2a7014baf9126dea0a6ce1bb4. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:44175: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:24.104678 23878 consensus_peers.cc:485] T 6a5addacc5e44689afcc942e217c0ec2 P d40aeb06cfcc4c3a897b40290d12a15b -> Peer 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175): Couldn't send request to peer 8795a3c2a7014baf9126dea0a6ce1bb4. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:44175: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:24.183323 23878 consensus_peers.cc:485] T abf6fde8492d459cb00bc324e8dbc66b P d40aeb06cfcc4c3a897b40290d12a15b -> Peer 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175): Couldn't send request to peer 8795a3c2a7014baf9126dea0a6ce1bb4. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:44175: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:24.206774 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 23730
W20240503 14:12:24.231576 24273 connection.cc:535] client connection to 127.22.242.66:46473 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:12:24.231634 24011 connection.cc:535] client connection to 127.22.242.66:46473 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:12:24.231905 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 23861
W20240503 14:12:24.231853 24011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:12:24.231884 24273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:12:24.232435 24140 consensus_peers.cc:485] T 9814abfe435b4bdf9fb65d65d03bc1b9 P c9a573f6faf64131b114af1e33c1b308 -> Peer 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175): Couldn't send request to peer 8795a3c2a7014baf9126dea0a6ce1bb4. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:44175: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:24.232533 24140 consensus_peers.cc:485] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 -> Peer 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175): Couldn't send request to peer 8795a3c2a7014baf9126dea0a6ce1bb4. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:44175: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:24.234743 24273 consensus_peers.cc:485] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 -> Peer d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473): Couldn't send request to peer d4eea540f3c241c1ba6290ca53769163. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.66:46473: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:24.234997 24011 consensus_peers.cc:485] T 5c55248037ca497bbc153b5933d4230e P 1e79cc5c647f476c98101f9e48b22b9e -> Peer d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473): Couldn't send request to peer d4eea540f3c241c1ba6290ca53769163. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.66:46473: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:24.239631 23984 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:12:24.253823 24273 consensus_peers.cc:485] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 -> Peer d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473): Couldn't send request to peer d4eea540f3c241c1ba6290ca53769163. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.66:46473: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:24.287130 24273 consensus_peers.cc:485] T 2cfc5cdabc784d1ea5f74b066f4e2be8 P ff53511345114c4ab97a9a8decef3a74 -> Peer d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473): Couldn't send request to peer d4eea540f3c241c1ba6290ca53769163. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.66:46473: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:24.305469 24140 consensus_peers.cc:485] T e99fc624e14a4e17a010e42e6c3960a2 P c9a573f6faf64131b114af1e33c1b308 -> Peer 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175): Couldn't send request to peer 8795a3c2a7014baf9126dea0a6ce1bb4. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:44175: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:24.307204 24273 consensus_peers.cc:485] T d31c3fbce3a74c32aff549e9d1ceda13 P ff53511345114c4ab97a9a8decef3a74 -> Peer d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473): Couldn't send request to peer d4eea540f3c241c1ba6290ca53769163. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.66:46473: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:24.311424 24273 consensus_peers.cc:485] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 -> Peer d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473): Couldn't send request to peer d4eea540f3c241c1ba6290ca53769163. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.66:46473: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:24.327116 24140 consensus_peers.cc:485] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 -> Peer 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175): Couldn't send request to peer 8795a3c2a7014baf9126dea0a6ce1bb4. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:44175: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:24.346814 24011 consensus_peers.cc:485] T f08228ffd3984f45afd068fab3eb9a43 P 1e79cc5c647f476c98101f9e48b22b9e -> Peer d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473): Couldn't send request to peer d4eea540f3c241c1ba6290ca53769163. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.66:46473: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:24.377895 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 23861
W20240503 14:12:24.390964 24011 consensus_peers.cc:485] T c1a414d5fd374961afe51938583bf66d P 1e79cc5c647f476c98101f9e48b22b9e -> Peer d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473): Couldn't send request to peer d4eea540f3c241c1ba6290ca53769163. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.66:46473: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:24.397405 24141 connection.cc:535] server connection from 127.22.242.67:46907 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:12:24.397747 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 23992
W20240503 14:12:24.398547 24139 consensus_peers.cc:485] T afec3382bc9043dda117d0c7c99c4816 P c9a573f6faf64131b114af1e33c1b308 -> Peer d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697): Couldn't send request to peer d40aeb06cfcc4c3a897b40290d12a15b. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.67:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:24.403570 24115 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:12:24.405647 24140 consensus_peers.cc:485] T f1c9262f9e154fe5b30c2ec66b26e4d5 P c9a573f6faf64131b114af1e33c1b308 -> Peer 8795a3c2a7014baf9126dea0a6ce1bb4 (127.22.242.65:44175): Couldn't send request to peer 8795a3c2a7014baf9126dea0a6ce1bb4. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:44175: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:24.516520 24139 consensus_peers.cc:485] T 174f75183c8f44c2a07ef326177d9d71 P c9a573f6faf64131b114af1e33c1b308 -> Peer d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697): Couldn't send request to peer d40aeb06cfcc4c3a897b40290d12a15b. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.67:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:24.516660 24139 consensus_peers.cc:485] T 666732f9b38f4a20a667bbb9880ece02 P c9a573f6faf64131b114af1e33c1b308 -> Peer d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697): Couldn't send request to peer d40aeb06cfcc4c3a897b40290d12a15b. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.67:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:24.532666 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 23992
W20240503 14:12:24.552992 24273 consensus_peers.cc:485] T 01f188dac6914137ba9607c3d7c468cb P ff53511345114c4ab97a9a8decef3a74 -> Peer d4eea540f3c241c1ba6290ca53769163 (127.22.242.66:46473): Couldn't send request to peer d4eea540f3c241c1ba6290ca53769163. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.66:46473: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:24.556924 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 24123
I20240503 14:12:24.562932 24245 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:12:24.584619 24273 consensus_peers.cc:485] T d70414f531ac463c9e89e2236d94250c P ff53511345114c4ab97a9a8decef3a74 -> Peer 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675): Couldn't send request to peer 1e79cc5c647f476c98101f9e48b22b9e. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.68:37675: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:24.618296 24270 consensus_peers.cc:485] T e0a3e437577742b1807b24a193ebc82b P ff53511345114c4ab97a9a8decef3a74 -> Peer d40aeb06cfcc4c3a897b40290d12a15b (127.22.242.67:37697): Couldn't send request to peer d40aeb06cfcc4c3a897b40290d12a15b. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.67:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:24.697362 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 24123
W20240503 14:12:24.712306 24273 consensus_peers.cc:485] T a78b4e88ed1e47908a4a2beae34a3ab7 P ff53511345114c4ab97a9a8decef3a74 -> Peer 1e79cc5c647f476c98101f9e48b22b9e (127.22.242.68:37675): Couldn't send request to peer 1e79cc5c647f476c98101f9e48b22b9e. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.68:37675: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:24.722380 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 24254
I20240503 14:12:24.729051 24377 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:24.863631 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 24254
W20240503 14:12:24.887063 23524 connection.cc:535] server connection from 127.22.242.70:42265 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:12:24.887363 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 23509
I20240503 14:12:24.888343 23570 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:25.035300 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 23509
2024-05-03T14:12:25Z chronyd exiting
[ OK ] LocationAwareRebalancingBasicTest.Basic (29925 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (29925 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240503 14:12:25.190973 23497 test_util.cc:268] Using random seed: 1327099416
I20240503 14:12:25.192967 23497 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:12:25.193044 23497 ts_itest-base.cc:113] --------------
I20240503 14:12:25.193101 23497 ts_itest-base.cc:114] 5 tablet servers
I20240503 14:12:25.193166 23497 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:12:25.193202 23497 ts_itest-base.cc:116] --------------
2024-05-03T14:12:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:12:25Z Disabled control of system clock
I20240503 14:12:25.228014 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.242.126:42887
--webserver_interface=127.22.242.126
--webserver_port=0
--builtin_ntp_servers=127.22.242.84:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.242.126:42887
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 14:12:25.557626 25887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:25.558009 25887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:25.558169 25887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:25.566290 25887 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:25.566468 25887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:12:25.566551 25887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:25.566639 25887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:12:25.566707 25887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:12:25.576022 25887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.242.126:42887
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.242.126:42887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.242.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:25.577422 25887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:25.579232 25887 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:25.590065 25892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:25.590030 25893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:25.590281 25895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:25.591018 25887 server_base.cc:1034] running on GCE node
I20240503 14:12:25.591878 25887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:25.593447 25887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:25.594652 25887 hybrid_clock.cc:648] HybridClock initialized: now 1714745545594601 us; error 51 us; skew 500 ppm
I20240503 14:12:25.595018 25887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:25.601729 25887 webserver.cc:460] Webserver started at http://127.22.242.126:35527/ using document root <none> and password file <none>
I20240503 14:12:25.602523 25887 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:25.602651 25887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:25.602979 25887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:25.605787 25887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d980099d81754c05aca48dae48dcd50d"
format_stamp: "Formatted at 2024-05-03 14:12:25 on dist-test-slave-r3pd"
I20240503 14:12:25.606597 25887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d980099d81754c05aca48dae48dcd50d"
format_stamp: "Formatted at 2024-05-03 14:12:25 on dist-test-slave-r3pd"
I20240503 14:12:25.611980 25887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 14:12:25.615463 25901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:25.617004 25887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:25.617211 25887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d980099d81754c05aca48dae48dcd50d"
format_stamp: "Formatted at 2024-05-03 14:12:25 on dist-test-slave-r3pd"
I20240503 14:12:25.617427 25887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:25.635030 25887 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:25.635931 25887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:25.636255 25887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:25.654098 25887 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.126:42887
I20240503 14:12:25.654109 25952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.126:42887 every 8 connection(s)
I20240503 14:12:25.655511 25887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:12:25.659114 25953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.660811 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 25887
I20240503 14:12:25.661042 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:12:25.670435 25953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d: Bootstrap starting.
I20240503 14:12:25.674073 25953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.675166 25953 log.cc:824] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:25.677839 25953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d: No bootstrap required, opened a new log
I20240503 14:12:25.684073 25953 raft_consensus.cc:348] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d980099d81754c05aca48dae48dcd50d" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 42887 } }
I20240503 14:12:25.684487 25953 raft_consensus.cc:374] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.684595 25953 raft_consensus.cc:729] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d980099d81754c05aca48dae48dcd50d, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.685199 25953 consensus_queue.cc:260] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d980099d81754c05aca48dae48dcd50d" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 42887 } }
I20240503 14:12:25.685406 25953 raft_consensus.cc:388] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:25.685547 25953 raft_consensus.cc:482] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:25.685719 25953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.687875 25953 raft_consensus.cc:504] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d980099d81754c05aca48dae48dcd50d" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 42887 } }
I20240503 14:12:25.688433 25953 leader_election.cc:302] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d980099d81754c05aca48dae48dcd50d; no voters:
I20240503 14:12:25.689002 25953 leader_election.cc:288] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:25.689136 25958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:25.690883 25958 raft_consensus.cc:686] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d [term 1 LEADER]: Becoming Leader. State: Replica: d980099d81754c05aca48dae48dcd50d, State: Running, Role: LEADER
I20240503 14:12:25.691300 25953 sys_catalog.cc:564] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:12:25.691519 25958 consensus_queue.cc:237] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d980099d81754c05aca48dae48dcd50d" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 42887 } }
I20240503 14:12:25.694206 25960 sys_catalog.cc:455] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d980099d81754c05aca48dae48dcd50d. Latest consensus state: current_term: 1 leader_uuid: "d980099d81754c05aca48dae48dcd50d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d980099d81754c05aca48dae48dcd50d" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 42887 } } }
I20240503 14:12:25.694582 25960 sys_catalog.cc:458] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:25.695147 25959 sys_catalog.cc:455] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d980099d81754c05aca48dae48dcd50d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d980099d81754c05aca48dae48dcd50d" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 42887 } } }
I20240503 14:12:25.695412 25959 sys_catalog.cc:458] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:25.695468 25967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:12:25.701112 25967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:12:25.710110 25967 catalog_manager.cc:1347] Generated new cluster ID: 0c282ff45b47427d938a4de7e564aa07
I20240503 14:12:25.710327 25967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:12:25.728679 25967 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:12:25.730244 25967 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:12:25.743310 25967 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d980099d81754c05aca48dae48dcd50d: Generated new TSK 0
I20240503 14:12:25.744262 25967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:12:25.758347 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.65:0
--local_ip_for_outbound_sockets=127.22.242.65
--webserver_interface=127.22.242.65
--webserver_port=0
--tserver_master_addrs=127.22.242.126:42887
--builtin_ntp_servers=127.22.242.84:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:12:26.125959 25977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:26.126245 25977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:26.126353 25977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:26.133914 25977 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:26.134034 25977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:26.134159 25977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.65
I20240503 14:12:26.143594 25977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.242.65
--webserver_port=0
--tserver_master_addrs=127.22.242.126:42887
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.65
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:26.144966 25977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:26.146670 25977 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:26.158236 25982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:26.158317 25983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:26.158295 25985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:26.159060 25977 server_base.cc:1034] running on GCE node
I20240503 14:12:26.159647 25977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:26.160616 25977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:26.161814 25977 hybrid_clock.cc:648] HybridClock initialized: now 1714745546161768 us; error 42 us; skew 500 ppm
I20240503 14:12:26.162113 25977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:26.164445 25977 webserver.cc:460] Webserver started at http://127.22.242.65:42827/ using document root <none> and password file <none>
I20240503 14:12:26.165174 25977 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:26.165302 25977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:26.165618 25977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:26.168272 25977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "00c5ed0c452c441aaee98a30558755fc"
format_stamp: "Formatted at 2024-05-03 14:12:26 on dist-test-slave-r3pd"
I20240503 14:12:26.169059 25977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "00c5ed0c452c441aaee98a30558755fc"
format_stamp: "Formatted at 2024-05-03 14:12:26 on dist-test-slave-r3pd"
I20240503 14:12:26.174223 25977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20240503 14:12:26.177651 25991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:26.179033 25977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:26.179280 25977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "00c5ed0c452c441aaee98a30558755fc"
format_stamp: "Formatted at 2024-05-03 14:12:26 on dist-test-slave-r3pd"
I20240503 14:12:26.179514 25977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:26.207702 25977 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:26.208703 25977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:26.208987 25977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:26.209988 25977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:26.211493 25977 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:26.211604 25977 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:26.211720 25977 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:26.211793 25977 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:26.244426 25977 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.65:33193
I20240503 14:12:26.244488 26103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.65:33193 every 8 connection(s)
I20240503 14:12:26.245765 25977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:12:26.256325 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 25977
I20240503 14:12:26.256515 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:12:26.259655 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.66:0
--local_ip_for_outbound_sockets=127.22.242.66
--webserver_interface=127.22.242.66
--webserver_port=0
--tserver_master_addrs=127.22.242.126:42887
--builtin_ntp_servers=127.22.242.84:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:26.264532 26104 heartbeater.cc:347] Connected to a master server at 127.22.242.126:42887
I20240503 14:12:26.264842 26104 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:26.265663 26104 heartbeater.cc:511] Master 127.22.242.126:42887 requested a full tablet report, sending...
I20240503 14:12:26.267691 25918 ts_manager.cc:194] Registered new tserver with Master: 00c5ed0c452c441aaee98a30558755fc (127.22.242.65:33193)
I20240503 14:12:26.269182 25918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.65:58323
W20240503 14:12:26.591713 26108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:26.592015 26108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:26.592206 26108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:26.599623 26108 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:26.599771 26108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:26.599939 26108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.66
I20240503 14:12:26.610316 26108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.242.66
--webserver_port=0
--tserver_master_addrs=127.22.242.126:42887
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.66
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:26.612082 26108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:26.614423 26108 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:26.628983 26113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:26.628643 26114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:26.628614 26116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:26.632601 26108 server_base.cc:1034] running on GCE node
I20240503 14:12:26.633415 26108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:26.634716 26108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:26.635957 26108 hybrid_clock.cc:648] HybridClock initialized: now 1714745546635879 us; error 68 us; skew 500 ppm
I20240503 14:12:26.636423 26108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:26.639556 26108 webserver.cc:460] Webserver started at http://127.22.242.66:44873/ using document root <none> and password file <none>
I20240503 14:12:26.640460 26108 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:26.640564 26108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:26.640959 26108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:26.644017 26108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f4057962926444bda6758777816581df"
format_stamp: "Formatted at 2024-05-03 14:12:26 on dist-test-slave-r3pd"
I20240503 14:12:26.645025 26108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f4057962926444bda6758777816581df"
format_stamp: "Formatted at 2024-05-03 14:12:26 on dist-test-slave-r3pd"
I20240503 14:12:26.650643 26108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:12:26.654889 26122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:26.657325 26108 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:26.657545 26108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f4057962926444bda6758777816581df"
format_stamp: "Formatted at 2024-05-03 14:12:26 on dist-test-slave-r3pd"
I20240503 14:12:26.657810 26108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:26.681663 26108 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:26.682502 26108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:26.682746 26108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:26.683670 26108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:26.685325 26108 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:26.685412 26108 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:26.685492 26108 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:26.685545 26108 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:26.717242 26108 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.66:45989
I20240503 14:12:26.717298 26234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.66:45989 every 8 connection(s)
I20240503 14:12:26.718540 26108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:12:26.723712 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 26108
I20240503 14:12:26.723907 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:12:26.727478 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.67:0
--local_ip_for_outbound_sockets=127.22.242.67
--webserver_interface=127.22.242.67
--webserver_port=0
--tserver_master_addrs=127.22.242.126:42887
--builtin_ntp_servers=127.22.242.84:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:26.731786 26235 heartbeater.cc:347] Connected to a master server at 127.22.242.126:42887
I20240503 14:12:26.732079 26235 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:26.732923 26235 heartbeater.cc:511] Master 127.22.242.126:42887 requested a full tablet report, sending...
I20240503 14:12:26.734313 25918 ts_manager.cc:194] Registered new tserver with Master: f4057962926444bda6758777816581df (127.22.242.66:45989)
I20240503 14:12:26.735064 25918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.66:33163
W20240503 14:12:27.059198 26238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:27.059494 26238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:27.059623 26238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:27.067188 26238 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:27.067312 26238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:27.067451 26238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.67
I20240503 14:12:27.076375 26238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.242.67
--webserver_port=0
--tserver_master_addrs=127.22.242.126:42887
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.67
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:27.077684 26238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:27.079371 26238 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:27.091342 26244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:27.091423 26243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:27.091363 26246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:27.092242 26238 server_base.cc:1034] running on GCE node
I20240503 14:12:27.092867 26238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:27.093811 26238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:27.095042 26238 hybrid_clock.cc:648] HybridClock initialized: now 1714745547094984 us; error 50 us; skew 500 ppm
I20240503 14:12:27.095386 26238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:27.097645 26238 webserver.cc:460] Webserver started at http://127.22.242.67:44725/ using document root <none> and password file <none>
I20240503 14:12:27.098397 26238 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:27.098506 26238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:27.098830 26238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:27.101527 26238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a62030f221e4497fb4557462c9d12a80"
format_stamp: "Formatted at 2024-05-03 14:12:27 on dist-test-slave-r3pd"
I20240503 14:12:27.102290 26238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a62030f221e4497fb4557462c9d12a80"
format_stamp: "Formatted at 2024-05-03 14:12:27 on dist-test-slave-r3pd"
I20240503 14:12:27.107517 26238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240503 14:12:27.110898 26252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:27.112293 26238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:27.112454 26238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a62030f221e4497fb4557462c9d12a80"
format_stamp: "Formatted at 2024-05-03 14:12:27 on dist-test-slave-r3pd"
I20240503 14:12:27.112648 26238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:27.127947 26238 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:27.128834 26238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:27.129105 26238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:27.130046 26238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:27.131489 26238 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:27.131577 26238 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:27.131695 26238 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:27.131769 26238 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:27.163553 26238 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.67:36487
I20240503 14:12:27.163626 26364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.67:36487 every 8 connection(s)
I20240503 14:12:27.165069 26238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:12:27.170475 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 26238
I20240503 14:12:27.170612 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:12:27.173838 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.68:0
--local_ip_for_outbound_sockets=127.22.242.68
--webserver_interface=127.22.242.68
--webserver_port=0
--tserver_master_addrs=127.22.242.126:42887
--builtin_ntp_servers=127.22.242.84:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:27.178040 26365 heartbeater.cc:347] Connected to a master server at 127.22.242.126:42887
I20240503 14:12:27.178331 26365 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:27.179158 26365 heartbeater.cc:511] Master 127.22.242.126:42887 requested a full tablet report, sending...
I20240503 14:12:27.180655 25918 ts_manager.cc:194] Registered new tserver with Master: a62030f221e4497fb4557462c9d12a80 (127.22.242.67:36487)
I20240503 14:12:27.181478 25918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.67:40367
I20240503 14:12:27.271926 26104 heartbeater.cc:503] Master 127.22.242.126:42887 was elected leader, sending a full tablet report...
W20240503 14:12:27.510519 26368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:27.510852 26368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:27.510988 26368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:27.518409 26368 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:27.518575 26368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:27.518754 26368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.68
I20240503 14:12:27.528044 26368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.242.68
--webserver_port=0
--tserver_master_addrs=127.22.242.126:42887
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.68
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:27.529475 26368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:27.531277 26368 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:27.543942 26374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:27.543889 26376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:27.543916 26373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:27.544742 26368 server_base.cc:1034] running on GCE node
I20240503 14:12:27.545394 26368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:27.546394 26368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:27.547607 26368 hybrid_clock.cc:648] HybridClock initialized: now 1714745547547551 us; error 55 us; skew 500 ppm
I20240503 14:12:27.547927 26368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:27.550372 26368 webserver.cc:460] Webserver started at http://127.22.242.68:33921/ using document root <none> and password file <none>
I20240503 14:12:27.551115 26368 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:27.551245 26368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:27.551573 26368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:27.554298 26368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "8ecc79a4291a42ee9213301505e7fd82"
format_stamp: "Formatted at 2024-05-03 14:12:27 on dist-test-slave-r3pd"
I20240503 14:12:27.555089 26368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "8ecc79a4291a42ee9213301505e7fd82"
format_stamp: "Formatted at 2024-05-03 14:12:27 on dist-test-slave-r3pd"
I20240503 14:12:27.561080 26368 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20240503 14:12:27.564679 26382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:27.566143 26368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240503 14:12:27.566355 26368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "8ecc79a4291a42ee9213301505e7fd82"
format_stamp: "Formatted at 2024-05-03 14:12:27 on dist-test-slave-r3pd"
I20240503 14:12:27.566576 26368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:27.627728 26368 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:27.628731 26368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:27.629019 26368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:27.630021 26368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:27.631523 26368 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:27.631623 26368 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:27.631747 26368 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:27.631819 26368 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:27.666568 26368 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.68:38931
I20240503 14:12:27.666620 26494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.68:38931 every 8 connection(s)
I20240503 14:12:27.667855 26368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:12:27.668913 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 26368
I20240503 14:12:27.669085 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:12:27.672463 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.69:0
--local_ip_for_outbound_sockets=127.22.242.69
--webserver_interface=127.22.242.69
--webserver_port=0
--tserver_master_addrs=127.22.242.126:42887
--builtin_ntp_servers=127.22.242.84:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:27.682435 26495 heartbeater.cc:347] Connected to a master server at 127.22.242.126:42887
I20240503 14:12:27.682786 26495 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:27.683640 26495 heartbeater.cc:511] Master 127.22.242.126:42887 requested a full tablet report, sending...
I20240503 14:12:27.685251 25918 ts_manager.cc:194] Registered new tserver with Master: 8ecc79a4291a42ee9213301505e7fd82 (127.22.242.68:38931)
I20240503 14:12:27.686478 25918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.68:48813
I20240503 14:12:27.737547 26235 heartbeater.cc:503] Master 127.22.242.126:42887 was elected leader, sending a full tablet report...
W20240503 14:12:28.009093 26498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:28.009402 26498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:28.009490 26498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:28.017520 26498 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:28.017645 26498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:28.017773 26498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.69
I20240503 14:12:28.027182 26498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.242.69
--webserver_port=0
--tserver_master_addrs=127.22.242.126:42887
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.69
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:28.028599 26498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:28.030310 26498 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:28.041980 26504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:28.042058 26503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:28.042014 26506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:28.042773 26498 server_base.cc:1034] running on GCE node
I20240503 14:12:28.043373 26498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:28.044322 26498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:28.045524 26498 hybrid_clock.cc:648] HybridClock initialized: now 1714745548045479 us; error 37 us; skew 500 ppm
I20240503 14:12:28.045850 26498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:28.048041 26498 webserver.cc:460] Webserver started at http://127.22.242.69:42805/ using document root <none> and password file <none>
I20240503 14:12:28.048822 26498 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:28.048936 26498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:28.049252 26498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:28.051784 26498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e58c6ffbcff3416683dedcda3fadc7c2"
format_stamp: "Formatted at 2024-05-03 14:12:28 on dist-test-slave-r3pd"
I20240503 14:12:28.052574 26498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e58c6ffbcff3416683dedcda3fadc7c2"
format_stamp: "Formatted at 2024-05-03 14:12:28 on dist-test-slave-r3pd"
I20240503 14:12:28.057546 26498 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240503 14:12:28.060853 26512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:28.062116 26498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:28.062274 26498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e58c6ffbcff3416683dedcda3fadc7c2"
format_stamp: "Formatted at 2024-05-03 14:12:28 on dist-test-slave-r3pd"
I20240503 14:12:28.062461 26498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:28.080982 26498 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:28.081830 26498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:28.082103 26498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:28.083036 26498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:28.084623 26498 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:28.084730 26498 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:28.084846 26498 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:28.084928 26498 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:28.117242 26498 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.69:35381
I20240503 14:12:28.117296 26624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.69:35381 every 8 connection(s)
I20240503 14:12:28.118614 26498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:12:28.127454 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 26498
I20240503 14:12:28.127637 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:12:28.130657 26625 heartbeater.cc:347] Connected to a master server at 127.22.242.126:42887
I20240503 14:12:28.130971 26625 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:28.131803 26625 heartbeater.cc:511] Master 127.22.242.126:42887 requested a full tablet report, sending...
I20240503 14:12:28.133136 25917 ts_manager.cc:194] Registered new tserver with Master: e58c6ffbcff3416683dedcda3fadc7c2 (127.22.242.69:35381)
I20240503 14:12:28.133975 25917 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.69:49241
I20240503 14:12:28.143836 23497 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 14:12:28.159070 25917 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55658:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 14:12:28.185298 26365 heartbeater.cc:503] Master 127.22.242.126:42887 was elected leader, sending a full tablet report...
I20240503 14:12:28.190765 26039 tablet_service.cc:1467] Processing CreateTablet for tablet 4f4a0a7f79f04345897bc3071b38b556 (DEFAULT_TABLE table=TestTable [id=5046c5b194254df9ab0b20e43b52093a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.190805 26300 tablet_service.cc:1467] Processing CreateTablet for tablet 4f4a0a7f79f04345897bc3071b38b556 (DEFAULT_TABLE table=TestTable [id=5046c5b194254df9ab0b20e43b52093a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.192371 26300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f4a0a7f79f04345897bc3071b38b556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.192360 26039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f4a0a7f79f04345897bc3071b38b556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.192184 26560 tablet_service.cc:1467] Processing CreateTablet for tablet 4f4a0a7f79f04345897bc3071b38b556 (DEFAULT_TABLE table=TestTable [id=5046c5b194254df9ab0b20e43b52093a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.193449 26560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f4a0a7f79f04345897bc3071b38b556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.203939 26643 tablet_bootstrap.cc:492] T 4f4a0a7f79f04345897bc3071b38b556 P e58c6ffbcff3416683dedcda3fadc7c2: Bootstrap starting.
I20240503 14:12:28.206709 26645 tablet_bootstrap.cc:492] T 4f4a0a7f79f04345897bc3071b38b556 P a62030f221e4497fb4557462c9d12a80: Bootstrap starting.
I20240503 14:12:28.206840 26644 tablet_bootstrap.cc:492] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc: Bootstrap starting.
I20240503 14:12:28.208618 26643 tablet_bootstrap.cc:654] T 4f4a0a7f79f04345897bc3071b38b556 P e58c6ffbcff3416683dedcda3fadc7c2: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.209852 26644 tablet_bootstrap.cc:654] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.209915 26643 log.cc:824] T 4f4a0a7f79f04345897bc3071b38b556 P e58c6ffbcff3416683dedcda3fadc7c2: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:28.210650 26645 tablet_bootstrap.cc:654] T 4f4a0a7f79f04345897bc3071b38b556 P a62030f221e4497fb4557462c9d12a80: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.210929 26644 log.cc:824] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:28.211710 26645 log.cc:824] T 4f4a0a7f79f04345897bc3071b38b556 P a62030f221e4497fb4557462c9d12a80: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:28.212975 26643 tablet_bootstrap.cc:492] T 4f4a0a7f79f04345897bc3071b38b556 P e58c6ffbcff3416683dedcda3fadc7c2: No bootstrap required, opened a new log
I20240503 14:12:28.213268 26643 ts_tablet_manager.cc:1367] T 4f4a0a7f79f04345897bc3071b38b556 P e58c6ffbcff3416683dedcda3fadc7c2: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20240503 14:12:28.213711 26644 tablet_bootstrap.cc:492] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc: No bootstrap required, opened a new log
I20240503 14:12:28.214083 26644 ts_tablet_manager.cc:1367] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240503 14:12:28.214386 26645 tablet_bootstrap.cc:492] T 4f4a0a7f79f04345897bc3071b38b556 P a62030f221e4497fb4557462c9d12a80: No bootstrap required, opened a new log
I20240503 14:12:28.214651 26645 ts_tablet_manager.cc:1367] T 4f4a0a7f79f04345897bc3071b38b556 P a62030f221e4497fb4557462c9d12a80: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20240503 14:12:28.219717 26643 raft_consensus.cc:348] T 4f4a0a7f79f04345897bc3071b38b556 P e58c6ffbcff3416683dedcda3fadc7c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c5ed0c452c441aaee98a30558755fc" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 33193 } } peers { permanent_uuid: "a62030f221e4497fb4557462c9d12a80" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 36487 } } peers { permanent_uuid: "e58c6ffbcff3416683dedcda3fadc7c2" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 35381 } }
I20240503 14:12:28.220091 26643 raft_consensus.cc:374] T 4f4a0a7f79f04345897bc3071b38b556 P e58c6ffbcff3416683dedcda3fadc7c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.220211 26643 raft_consensus.cc:729] T 4f4a0a7f79f04345897bc3071b38b556 P e58c6ffbcff3416683dedcda3fadc7c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58c6ffbcff3416683dedcda3fadc7c2, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.220952 26643 consensus_queue.cc:260] T 4f4a0a7f79f04345897bc3071b38b556 P e58c6ffbcff3416683dedcda3fadc7c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c5ed0c452c441aaee98a30558755fc" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 33193 } } peers { permanent_uuid: "a62030f221e4497fb4557462c9d12a80" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 36487 } } peers { permanent_uuid: "e58c6ffbcff3416683dedcda3fadc7c2" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 35381 } }
I20240503 14:12:28.222792 26643 ts_tablet_manager.cc:1398] T 4f4a0a7f79f04345897bc3071b38b556 P e58c6ffbcff3416683dedcda3fadc7c2: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20240503 14:12:28.223296 26625 heartbeater.cc:503] Master 127.22.242.126:42887 was elected leader, sending a full tablet report...
I20240503 14:12:28.223639 26644 raft_consensus.cc:348] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c5ed0c452c441aaee98a30558755fc" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 33193 } } peers { permanent_uuid: "a62030f221e4497fb4557462c9d12a80" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 36487 } } peers { permanent_uuid: "e58c6ffbcff3416683dedcda3fadc7c2" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 35381 } }
I20240503 14:12:28.223639 26645 raft_consensus.cc:348] T 4f4a0a7f79f04345897bc3071b38b556 P a62030f221e4497fb4557462c9d12a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c5ed0c452c441aaee98a30558755fc" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 33193 } } peers { permanent_uuid: "a62030f221e4497fb4557462c9d12a80" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 36487 } } peers { permanent_uuid: "e58c6ffbcff3416683dedcda3fadc7c2" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 35381 } }
I20240503 14:12:28.224088 26645 raft_consensus.cc:374] T 4f4a0a7f79f04345897bc3071b38b556 P a62030f221e4497fb4557462c9d12a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.224088 26644 raft_consensus.cc:374] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.224229 26644 raft_consensus.cc:729] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c5ed0c452c441aaee98a30558755fc, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.224260 26645 raft_consensus.cc:729] T 4f4a0a7f79f04345897bc3071b38b556 P a62030f221e4497fb4557462c9d12a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62030f221e4497fb4557462c9d12a80, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.224960 26645 consensus_queue.cc:260] T 4f4a0a7f79f04345897bc3071b38b556 P a62030f221e4497fb4557462c9d12a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c5ed0c452c441aaee98a30558755fc" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 33193 } } peers { permanent_uuid: "a62030f221e4497fb4557462c9d12a80" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 36487 } } peers { permanent_uuid: "e58c6ffbcff3416683dedcda3fadc7c2" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 35381 } }
I20240503 14:12:28.224961 26644 consensus_queue.cc:260] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c5ed0c452c441aaee98a30558755fc" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 33193 } } peers { permanent_uuid: "a62030f221e4497fb4557462c9d12a80" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 36487 } } peers { permanent_uuid: "e58c6ffbcff3416683dedcda3fadc7c2" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 35381 } }
I20240503 14:12:28.226956 26645 ts_tablet_manager.cc:1398] T 4f4a0a7f79f04345897bc3071b38b556 P a62030f221e4497fb4557462c9d12a80: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20240503 14:12:28.228720 26644 ts_tablet_manager.cc:1398] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
W20240503 14:12:28.248740 26105 tablet.cc:2369] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:28.250205 26650 raft_consensus.cc:482] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.250453 26650 raft_consensus.cc:504] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c5ed0c452c441aaee98a30558755fc" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 33193 } } peers { permanent_uuid: "a62030f221e4497fb4557462c9d12a80" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 36487 } } peers { permanent_uuid: "e58c6ffbcff3416683dedcda3fadc7c2" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 35381 } }
I20240503 14:12:28.252578 26650 leader_election.cc:288] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a62030f221e4497fb4557462c9d12a80 (127.22.242.67:36487), e58c6ffbcff3416683dedcda3fadc7c2 (127.22.242.69:35381)
I20240503 14:12:28.259716 26580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f4a0a7f79f04345897bc3071b38b556" candidate_uuid: "00c5ed0c452c441aaee98a30558755fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e58c6ffbcff3416683dedcda3fadc7c2" is_pre_election: true
I20240503 14:12:28.259735 26320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f4a0a7f79f04345897bc3071b38b556" candidate_uuid: "00c5ed0c452c441aaee98a30558755fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62030f221e4497fb4557462c9d12a80" is_pre_election: true
I20240503 14:12:28.260222 26320 raft_consensus.cc:2454] T 4f4a0a7f79f04345897bc3071b38b556 P a62030f221e4497fb4557462c9d12a80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c5ed0c452c441aaee98a30558755fc in term 0.
I20240503 14:12:28.260233 26580 raft_consensus.cc:2454] T 4f4a0a7f79f04345897bc3071b38b556 P e58c6ffbcff3416683dedcda3fadc7c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c5ed0c452c441aaee98a30558755fc in term 0.
I20240503 14:12:28.260854 25995 leader_election.cc:302] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c5ed0c452c441aaee98a30558755fc, a62030f221e4497fb4557462c9d12a80; no voters:
I20240503 14:12:28.261234 26650 raft_consensus.cc:2781] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:28.261374 26650 raft_consensus.cc:482] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:28.261484 26650 raft_consensus.cc:3037] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.263489 26650 raft_consensus.cc:504] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c5ed0c452c441aaee98a30558755fc" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 33193 } } peers { permanent_uuid: "a62030f221e4497fb4557462c9d12a80" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 36487 } } peers { permanent_uuid: "e58c6ffbcff3416683dedcda3fadc7c2" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 35381 } }
I20240503 14:12:28.264075 26650 leader_election.cc:288] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [CANDIDATE]: Term 1 election: Requested vote from peers a62030f221e4497fb4557462c9d12a80 (127.22.242.67:36487), e58c6ffbcff3416683dedcda3fadc7c2 (127.22.242.69:35381)
I20240503 14:12:28.264564 26320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f4a0a7f79f04345897bc3071b38b556" candidate_uuid: "00c5ed0c452c441aaee98a30558755fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62030f221e4497fb4557462c9d12a80"
I20240503 14:12:28.264576 26580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f4a0a7f79f04345897bc3071b38b556" candidate_uuid: "00c5ed0c452c441aaee98a30558755fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e58c6ffbcff3416683dedcda3fadc7c2"
I20240503 14:12:28.264755 26580 raft_consensus.cc:3037] T 4f4a0a7f79f04345897bc3071b38b556 P e58c6ffbcff3416683dedcda3fadc7c2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.264762 26320 raft_consensus.cc:3037] T 4f4a0a7f79f04345897bc3071b38b556 P a62030f221e4497fb4557462c9d12a80 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.266916 26320 raft_consensus.cc:2454] T 4f4a0a7f79f04345897bc3071b38b556 P a62030f221e4497fb4557462c9d12a80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c5ed0c452c441aaee98a30558755fc in term 1.
I20240503 14:12:28.266932 26580 raft_consensus.cc:2454] T 4f4a0a7f79f04345897bc3071b38b556 P e58c6ffbcff3416683dedcda3fadc7c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c5ed0c452c441aaee98a30558755fc in term 1.
I20240503 14:12:28.267310 25995 leader_election.cc:302] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c5ed0c452c441aaee98a30558755fc, a62030f221e4497fb4557462c9d12a80; no voters:
I20240503 14:12:28.267598 26650 raft_consensus.cc:2781] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:28.268000 26650 raft_consensus.cc:686] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [term 1 LEADER]: Becoming Leader. State: Replica: 00c5ed0c452c441aaee98a30558755fc, State: Running, Role: LEADER
I20240503 14:12:28.268563 26650 consensus_queue.cc:237] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c5ed0c452c441aaee98a30558755fc" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 33193 } } peers { permanent_uuid: "a62030f221e4497fb4557462c9d12a80" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 36487 } } peers { permanent_uuid: "e58c6ffbcff3416683dedcda3fadc7c2" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 35381 } }
I20240503 14:12:28.273015 25916 catalog_manager.cc:5484] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c5ed0c452c441aaee98a30558755fc (127.22.242.65). New cstate: current_term: 1 leader_uuid: "00c5ed0c452c441aaee98a30558755fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c5ed0c452c441aaee98a30558755fc" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62030f221e4497fb4557462c9d12a80" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 36487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e58c6ffbcff3416683dedcda3fadc7c2" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 35381 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:28.286803 23497 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 14:12:28.288972 23497 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 00c5ed0c452c441aaee98a30558755fc to finish bootstrapping
I20240503 14:12:28.297695 23497 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f4057962926444bda6758777816581df to finish bootstrapping
I20240503 14:12:28.306754 23497 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a62030f221e4497fb4557462c9d12a80 to finish bootstrapping
I20240503 14:12:28.313941 23497 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8ecc79a4291a42ee9213301505e7fd82 to finish bootstrapping
I20240503 14:12:28.322337 23497 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e58c6ffbcff3416683dedcda3fadc7c2 to finish bootstrapping
W20240503 14:12:28.370267 26626 tablet.cc:2369] T 4f4a0a7f79f04345897bc3071b38b556 P e58c6ffbcff3416683dedcda3fadc7c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:12:28.417583 26366 tablet.cc:2369] T 4f4a0a7f79f04345897bc3071b38b556 P a62030f221e4497fb4557462c9d12a80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:28.647943 26650 consensus_queue.cc:1035] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e58c6ffbcff3416683dedcda3fadc7c2" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 35381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:28.656976 26654 consensus_queue.cc:1035] T 4f4a0a7f79f04345897bc3071b38b556 P 00c5ed0c452c441aaee98a30558755fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62030f221e4497fb4557462c9d12a80" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 36487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:28.689074 26495 heartbeater.cc:503] Master 127.22.242.126:42887 was elected leader, sending a full tablet report...
I20240503 14:12:28.825362 26430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:28.828063 26039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:28.831521 26300 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:28.834614 26560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:28.849246 26170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:28.999648 26039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:29.010344 26300 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:29.017139 26430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:29.024737 26560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:29.058672 26170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:29.659265 26300 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:29.662477 26560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:29.664481 26430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:29.668677 26170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:29.676487 26039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:29.840279 26039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:29.851014 26430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:29.853739 26170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:29.855526 26560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:29.861377 26300 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:30.027468 26430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:30.032581 26300 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:30.033634 26560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:30.035136 26039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:30.048462 26170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:30.200523 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 25977
I20240503 14:12:30.202361 26099 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:30.301077 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 25977
I20240503 14:12:30.316594 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 26108
I20240503 14:12:30.317724 26230 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:30.408296 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 26108
I20240503 14:12:30.426497 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 26238
I20240503 14:12:30.427531 26359 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:30.521499 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 26238
I20240503 14:12:30.539439 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 26368
I20240503 14:12:30.540444 26490 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:30.631052 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 26368
I20240503 14:12:30.648640 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 26498
I20240503 14:12:30.649597 26619 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:30.742122 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 26498
W20240503 14:12:30.759286 25903 connection.cc:535] server connection from 127.22.242.69:49241 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:12:30.759529 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 25887
I20240503 14:12:30.760681 25948 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:30.876204 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 25887
2024-05-03T14:12:30Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (5728 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5728 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/3
I20240503 14:12:30.919875 23497 test_util.cc:268] Using random seed: 1332828328
I20240503 14:12:30.921808 23497 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:12:30.921875 23497 ts_itest-base.cc:113] --------------
I20240503 14:12:30.921926 23497 ts_itest-base.cc:114] 7 tablet servers
I20240503 14:12:30.921967 23497 ts_itest-base.cc:115] 2 replicas per TS
I20240503 14:12:30.922016 23497 ts_itest-base.cc:116] --------------
2024-05-03T14:12:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:12:30Z Disabled control of system clock
I20240503 14:12:30.956336 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.242.126:45965
--webserver_interface=127.22.242.126
--webserver_port=0
--builtin_ntp_servers=127.22.242.84:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.242.126:45965
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 14:12:31.284636 26851 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 14:12:31.284982 26851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:31.285084 26851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:31.285202 26851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:31.292807 26851 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:31.292932 26851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:12:31.293041 26851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:31.293103 26851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:12:31.293205 26851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:12:31.302423 26851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.242.126:45965
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.242.126:45965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.242.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:31.303799 26851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:31.305559 26851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:31.315873 26857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:31.315950 26856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:31.315917 26859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:31.316957 26851 server_base.cc:1034] running on GCE node
I20240503 14:12:31.317732 26851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:31.319072 26851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:31.320276 26851 hybrid_clock.cc:648] HybridClock initialized: now 1714745551320210 us; error 35 us; skew 500 ppm
I20240503 14:12:31.320633 26851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:31.341759 26851 webserver.cc:460] Webserver started at http://127.22.242.126:41275/ using document root <none> and password file <none>
I20240503 14:12:31.342578 26851 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:31.342715 26851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:31.343036 26851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:31.345789 26851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "cf0ddf686eec42e5897a2ea24534acfb"
format_stamp: "Formatted at 2024-05-03 14:12:31 on dist-test-slave-r3pd"
I20240503 14:12:31.346549 26851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "cf0ddf686eec42e5897a2ea24534acfb"
format_stamp: "Formatted at 2024-05-03 14:12:31 on dist-test-slave-r3pd"
I20240503 14:12:31.351842 26851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:12:31.355199 26865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:31.356588 26851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:31.356755 26851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
uuid: "cf0ddf686eec42e5897a2ea24534acfb"
format_stamp: "Formatted at 2024-05-03 14:12:31 on dist-test-slave-r3pd"
I20240503 14:12:31.356937 26851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:31.381485 26851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:31.382411 26851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:31.382719 26851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:31.400980 26851 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.126:45965
I20240503 14:12:31.401005 26916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.126:45965 every 8 connection(s)
I20240503 14:12:31.402415 26851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:12:31.406275 26917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.409547 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 26851
I20240503 14:12:31.409775 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:12:31.418480 26917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb: Bootstrap starting.
I20240503 14:12:31.422013 26917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.423483 26917 log.cc:824] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:31.426375 26917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb: No bootstrap required, opened a new log
I20240503 14:12:31.432762 26917 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0ddf686eec42e5897a2ea24534acfb" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 45965 } }
I20240503 14:12:31.433223 26917 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.433367 26917 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf0ddf686eec42e5897a2ea24534acfb, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.434067 26917 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0ddf686eec42e5897a2ea24534acfb" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 45965 } }
I20240503 14:12:31.434266 26917 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:31.434362 26917 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:31.434489 26917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:31.436600 26917 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0ddf686eec42e5897a2ea24534acfb" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 45965 } }
I20240503 14:12:31.437052 26917 leader_election.cc:302] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf0ddf686eec42e5897a2ea24534acfb; no voters:
I20240503 14:12:31.437584 26917 leader_election.cc:288] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:31.437721 26922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:31.438565 26922 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb [term 1 LEADER]: Becoming Leader. State: Replica: cf0ddf686eec42e5897a2ea24534acfb, State: Running, Role: LEADER
I20240503 14:12:31.439121 26922 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0ddf686eec42e5897a2ea24534acfb" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 45965 } }
I20240503 14:12:31.439703 26917 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:12:31.441812 26924 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf0ddf686eec42e5897a2ea24534acfb. Latest consensus state: current_term: 1 leader_uuid: "cf0ddf686eec42e5897a2ea24534acfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0ddf686eec42e5897a2ea24534acfb" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 45965 } } }
I20240503 14:12:31.441780 26923 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf0ddf686eec42e5897a2ea24534acfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0ddf686eec42e5897a2ea24534acfb" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 45965 } } }
I20240503 14:12:31.442098 26924 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:31.442180 26923 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:31.442894 26927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:12:31.447242 26927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:12:31.456140 26927 catalog_manager.cc:1347] Generated new cluster ID: 581920d1c8dd41a4897798bdc9de9357
I20240503 14:12:31.456286 26927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:12:31.474018 26927 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:12:31.475396 26927 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:12:31.491669 26927 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cf0ddf686eec42e5897a2ea24534acfb: Generated new TSK 0
I20240503 14:12:31.492617 26927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:12:31.507118 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.65:0
--local_ip_for_outbound_sockets=127.22.242.65
--webserver_interface=127.22.242.65
--webserver_port=0
--tserver_master_addrs=127.22.242.126:45965
--builtin_ntp_servers=127.22.242.84:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:12:31.845703 26941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:31.846001 26941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:31.846091 26941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:31.853708 26941 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:31.853837 26941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:31.853962 26941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.65
I20240503 14:12:31.863059 26941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.242.65
--webserver_port=0
--tserver_master_addrs=127.22.242.126:45965
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.65
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:31.864331 26941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:31.865989 26941 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:31.877183 26946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:31.877328 26947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:31.877476 26949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:31.878324 26941 server_base.cc:1034] running on GCE node
I20240503 14:12:31.878902 26941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:31.879926 26941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:31.881126 26941 hybrid_clock.cc:648] HybridClock initialized: now 1714745551881059 us; error 59 us; skew 500 ppm
I20240503 14:12:31.881448 26941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:31.883666 26941 webserver.cc:460] Webserver started at http://127.22.242.65:45097/ using document root <none> and password file <none>
I20240503 14:12:31.884460 26941 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:31.884581 26941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:31.884897 26941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:31.887624 26941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "820e18ddcc1e4d53821d43bc32652bc9"
format_stamp: "Formatted at 2024-05-03 14:12:31 on dist-test-slave-r3pd"
I20240503 14:12:31.888494 26941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "820e18ddcc1e4d53821d43bc32652bc9"
format_stamp: "Formatted at 2024-05-03 14:12:31 on dist-test-slave-r3pd"
I20240503 14:12:31.893739 26941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 14:12:31.897104 26955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:31.898415 26941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:31.898572 26941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "820e18ddcc1e4d53821d43bc32652bc9"
format_stamp: "Formatted at 2024-05-03 14:12:31 on dist-test-slave-r3pd"
I20240503 14:12:31.898767 26941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:31.921140 26941 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:31.922164 26941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:31.922544 26941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:31.923454 26941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:31.924993 26941 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:31.925081 26941 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:31.925165 26941 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:31.925220 26941 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:31.956689 26941 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.65:37969
I20240503 14:12:31.956761 27067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.65:37969 every 8 connection(s)
I20240503 14:12:31.957984 26941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:12:31.961216 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 26941
I20240503 14:12:31.961413 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:12:31.964604 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.66:0
--local_ip_for_outbound_sockets=127.22.242.66
--webserver_interface=127.22.242.66
--webserver_port=0
--tserver_master_addrs=127.22.242.126:45965
--builtin_ntp_servers=127.22.242.84:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:31.971176 27068 heartbeater.cc:347] Connected to a master server at 127.22.242.126:45965
I20240503 14:12:31.971478 27068 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:31.972311 27068 heartbeater.cc:511] Master 127.22.242.126:45965 requested a full tablet report, sending...
I20240503 14:12:31.974274 26882 ts_manager.cc:194] Registered new tserver with Master: 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
I20240503 14:12:31.975733 26882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.65:50837
W20240503 14:12:32.298344 27072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:32.298642 27072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:32.298801 27072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:32.306558 27072 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:32.306689 27072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:32.306864 27072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.66
I20240503 14:12:32.316177 27072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.242.66
--webserver_port=0
--tserver_master_addrs=127.22.242.126:45965
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.66
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:32.317497 27072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:32.319243 27072 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:32.331104 27078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:32.331211 27077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:32.331218 27080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:32.331916 27072 server_base.cc:1034] running on GCE node
I20240503 14:12:32.332584 27072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:32.333582 27072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:32.334798 27072 hybrid_clock.cc:648] HybridClock initialized: now 1714745552334741 us; error 47 us; skew 500 ppm
I20240503 14:12:32.335120 27072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:32.337428 27072 webserver.cc:460] Webserver started at http://127.22.242.66:35649/ using document root <none> and password file <none>
I20240503 14:12:32.338183 27072 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:32.338303 27072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:32.338627 27072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:32.341437 27072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "305daa9a4d874d9db0e5378e5d27b617"
format_stamp: "Formatted at 2024-05-03 14:12:32 on dist-test-slave-r3pd"
I20240503 14:12:32.342224 27072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "305daa9a4d874d9db0e5378e5d27b617"
format_stamp: "Formatted at 2024-05-03 14:12:32 on dist-test-slave-r3pd"
I20240503 14:12:32.347556 27072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:12:32.351125 27086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:32.352527 27072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:32.352725 27072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "305daa9a4d874d9db0e5378e5d27b617"
format_stamp: "Formatted at 2024-05-03 14:12:32 on dist-test-slave-r3pd"
I20240503 14:12:32.352917 27072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:32.366559 27072 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:32.367404 27072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:32.367683 27072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:32.368690 27072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:32.370131 27072 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:32.370230 27072 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:32.370361 27072 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:32.370438 27072 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:32.403463 27072 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.66:38751
I20240503 14:12:32.403506 27198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.66:38751 every 8 connection(s)
I20240503 14:12:32.404891 27072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:12:32.407784 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 27072
I20240503 14:12:32.407950 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:12:32.410909 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.67:0
--local_ip_for_outbound_sockets=127.22.242.67
--webserver_interface=127.22.242.67
--webserver_port=0
--tserver_master_addrs=127.22.242.126:45965
--builtin_ntp_servers=127.22.242.84:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:32.428139 27199 heartbeater.cc:347] Connected to a master server at 127.22.242.126:45965
I20240503 14:12:32.428535 27199 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:32.429630 27199 heartbeater.cc:511] Master 127.22.242.126:45965 requested a full tablet report, sending...
I20240503 14:12:32.431272 26882 ts_manager.cc:194] Registered new tserver with Master: 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:32.432055 26882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.66:58889
W20240503 14:12:32.747871 27202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:32.748242 27202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:32.748399 27202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:32.755813 27202 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:32.755936 27202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:32.756096 27202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.67
I20240503 14:12:32.765571 27202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.242.67
--webserver_port=0
--tserver_master_addrs=127.22.242.126:45965
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.67
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:32.766891 27202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:32.768756 27202 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:32.780114 27207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:32.780081 27208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:32.780375 27210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:32.781343 27202 server_base.cc:1034] running on GCE node
I20240503 14:12:32.781924 27202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:32.782861 27202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:32.784049 27202 hybrid_clock.cc:648] HybridClock initialized: now 1714745552784004 us; error 37 us; skew 500 ppm
I20240503 14:12:32.784394 27202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:32.786648 27202 webserver.cc:460] Webserver started at http://127.22.242.67:35111/ using document root <none> and password file <none>
I20240503 14:12:32.787386 27202 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:32.787488 27202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:32.787792 27202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:32.790381 27202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f6c292faf79e49edabdbc7a310a30503"
format_stamp: "Formatted at 2024-05-03 14:12:32 on dist-test-slave-r3pd"
I20240503 14:12:32.791142 27202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f6c292faf79e49edabdbc7a310a30503"
format_stamp: "Formatted at 2024-05-03 14:12:32 on dist-test-slave-r3pd"
I20240503 14:12:32.796129 27202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 14:12:32.799515 27216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:32.800920 27202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240503 14:12:32.801102 27202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f6c292faf79e49edabdbc7a310a30503"
format_stamp: "Formatted at 2024-05-03 14:12:32 on dist-test-slave-r3pd"
I20240503 14:12:32.801312 27202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:32.826925 27202 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:32.827816 27202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:32.828116 27202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:32.829141 27202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:32.830674 27202 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:32.830775 27202 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:32.830896 27202 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:32.830974 27202 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:32.863317 27202 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.67:45423
I20240503 14:12:32.863368 27328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.67:45423 every 8 connection(s)
I20240503 14:12:32.864624 27202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:12:32.869726 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 27202
I20240503 14:12:32.869918 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:12:32.873070 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.68:0
--local_ip_for_outbound_sockets=127.22.242.68
--webserver_interface=127.22.242.68
--webserver_port=0
--tserver_master_addrs=127.22.242.126:45965
--builtin_ntp_servers=127.22.242.84:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:32.878005 27329 heartbeater.cc:347] Connected to a master server at 127.22.242.126:45965
I20240503 14:12:32.878355 27329 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:32.879177 27329 heartbeater.cc:511] Master 127.22.242.126:45965 requested a full tablet report, sending...
I20240503 14:12:32.880587 26882 ts_manager.cc:194] Registered new tserver with Master: f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423)
I20240503 14:12:32.881357 26882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.67:45805
I20240503 14:12:32.978693 27068 heartbeater.cc:503] Master 127.22.242.126:45965 was elected leader, sending a full tablet report...
W20240503 14:12:33.203861 27332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:33.204236 27332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:33.204350 27332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:33.212152 27332 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:33.212339 27332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:33.212471 27332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.68
I20240503 14:12:33.221846 27332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.242.68
--webserver_port=0
--tserver_master_addrs=127.22.242.126:45965
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.68
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:33.223194 27332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:33.225030 27332 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:33.236817 27340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:33.236879 27337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:33.236902 27338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:33.237953 27332 server_base.cc:1034] running on GCE node
I20240503 14:12:33.238538 27332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:33.239500 27332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:33.240691 27332 hybrid_clock.cc:648] HybridClock initialized: now 1714745553240628 us; error 54 us; skew 500 ppm
I20240503 14:12:33.241015 27332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:33.243283 27332 webserver.cc:460] Webserver started at http://127.22.242.68:39461/ using document root <none> and password file <none>
I20240503 14:12:33.244025 27332 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:33.244138 27332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:33.244505 27332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:33.247159 27332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4165ee79ee03466981e067d998339611"
format_stamp: "Formatted at 2024-05-03 14:12:33 on dist-test-slave-r3pd"
I20240503 14:12:33.247972 27332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4165ee79ee03466981e067d998339611"
format_stamp: "Formatted at 2024-05-03 14:12:33 on dist-test-slave-r3pd"
I20240503 14:12:33.253350 27332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240503 14:12:33.256788 27346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:33.258155 27332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240503 14:12:33.258342 27332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4165ee79ee03466981e067d998339611"
format_stamp: "Formatted at 2024-05-03 14:12:33 on dist-test-slave-r3pd"
I20240503 14:12:33.258523 27332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:33.276881 27332 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:33.277710 27332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:33.277959 27332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:33.278847 27332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:33.280264 27332 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:33.280349 27332 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:33.280431 27332 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:33.280485 27332 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:33.312644 27332 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.68:41205
I20240503 14:12:33.312695 27458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.68:41205 every 8 connection(s)
I20240503 14:12:33.313949 27332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:12:33.317270 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 27332
I20240503 14:12:33.317451 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:12:33.320736 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.69:0
--local_ip_for_outbound_sockets=127.22.242.69
--webserver_interface=127.22.242.69
--webserver_port=0
--tserver_master_addrs=127.22.242.126:45965
--builtin_ntp_servers=127.22.242.84:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:33.327433 27459 heartbeater.cc:347] Connected to a master server at 127.22.242.126:45965
I20240503 14:12:33.327751 27459 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:33.328640 27459 heartbeater.cc:511] Master 127.22.242.126:45965 requested a full tablet report, sending...
I20240503 14:12:33.330030 26882 ts_manager.cc:194] Registered new tserver with Master: 4165ee79ee03466981e067d998339611 (127.22.242.68:41205)
I20240503 14:12:33.330814 26882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.68:44097
I20240503 14:12:33.435591 27199 heartbeater.cc:503] Master 127.22.242.126:45965 was elected leader, sending a full tablet report...
W20240503 14:12:33.675098 27462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:33.675405 27462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:33.675493 27462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:33.682950 27462 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:33.683090 27462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:33.683207 27462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.69
I20240503 14:12:33.692338 27462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.242.69
--webserver_port=0
--tserver_master_addrs=127.22.242.126:45965
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.69
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:33.693668 27462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:33.695430 27462 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:33.707510 27470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:33.707542 27467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:33.707460 27468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:33.708525 27462 server_base.cc:1034] running on GCE node
I20240503 14:12:33.709127 27462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:33.710096 27462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:33.711308 27462 hybrid_clock.cc:648] HybridClock initialized: now 1714745553711261 us; error 38 us; skew 500 ppm
I20240503 14:12:33.711606 27462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:33.713928 27462 webserver.cc:460] Webserver started at http://127.22.242.69:44355/ using document root <none> and password file <none>
I20240503 14:12:33.714654 27462 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:33.714751 27462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:33.715037 27462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:33.718173 27462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "cbb4bf53151a4061a81cf5bacf084516"
format_stamp: "Formatted at 2024-05-03 14:12:33 on dist-test-slave-r3pd"
I20240503 14:12:33.719316 27462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "cbb4bf53151a4061a81cf5bacf084516"
format_stamp: "Formatted at 2024-05-03 14:12:33 on dist-test-slave-r3pd"
I20240503 14:12:33.726253 27462 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20240503 14:12:33.729817 27476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:33.731176 27462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:12:33.731348 27462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "cbb4bf53151a4061a81cf5bacf084516"
format_stamp: "Formatted at 2024-05-03 14:12:33 on dist-test-slave-r3pd"
I20240503 14:12:33.731549 27462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:33.755456 27462 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:33.756424 27462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:33.756711 27462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:33.757701 27462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:33.759178 27462 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:33.759289 27462 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:33.759404 27462 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:33.759476 27462 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:33.791090 27462 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.69:39081
I20240503 14:12:33.791138 27588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.69:39081 every 8 connection(s)
I20240503 14:12:33.792409 27462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:12:33.798942 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 27462
I20240503 14:12:33.799160 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:12:33.802804 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.70:0
--local_ip_for_outbound_sockets=127.22.242.70
--webserver_interface=127.22.242.70
--webserver_port=0
--tserver_master_addrs=127.22.242.126:45965
--builtin_ntp_servers=127.22.242.84:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:33.806334 27589 heartbeater.cc:347] Connected to a master server at 127.22.242.126:45965
I20240503 14:12:33.806717 27589 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:33.807796 27589 heartbeater.cc:511] Master 127.22.242.126:45965 requested a full tablet report, sending...
I20240503 14:12:33.809445 26882 ts_manager.cc:194] Registered new tserver with Master: cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69:39081)
I20240503 14:12:33.810191 26882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.69:53611
I20240503 14:12:33.883848 27329 heartbeater.cc:503] Master 127.22.242.126:45965 was elected leader, sending a full tablet report...
W20240503 14:12:34.139103 27592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:34.139403 27592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:34.139492 27592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:34.147140 27592 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:34.147276 27592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:34.147401 27592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.70
I20240503 14:12:34.156598 27592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.242.70
--webserver_port=0
--tserver_master_addrs=127.22.242.126:45965
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.70
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:34.157938 27592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:34.159642 27592 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:34.171231 27598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:34.171270 27600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:34.171195 27597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:34.171996 27592 server_base.cc:1034] running on GCE node
I20240503 14:12:34.172587 27592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:34.173477 27592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:34.174685 27592 hybrid_clock.cc:648] HybridClock initialized: now 1714745554174641 us; error 40 us; skew 500 ppm
I20240503 14:12:34.174998 27592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:34.177189 27592 webserver.cc:460] Webserver started at http://127.22.242.70:41231/ using document root <none> and password file <none>
I20240503 14:12:34.177891 27592 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:34.178012 27592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:34.178323 27592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:34.183107 27592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "89a56aa776934c4582bfbdb0622b0cb3"
format_stamp: "Formatted at 2024-05-03 14:12:34 on dist-test-slave-r3pd"
I20240503 14:12:34.183917 27592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "89a56aa776934c4582bfbdb0622b0cb3"
format_stamp: "Formatted at 2024-05-03 14:12:34 on dist-test-slave-r3pd"
I20240503 14:12:34.189298 27592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240503 14:12:34.192762 27606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:34.194120 27592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240503 14:12:34.194284 27592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "89a56aa776934c4582bfbdb0622b0cb3"
format_stamp: "Formatted at 2024-05-03 14:12:34 on dist-test-slave-r3pd"
I20240503 14:12:34.194484 27592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:34.215703 27592 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:34.216683 27592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:34.216980 27592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:34.217964 27592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:34.219496 27592 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:34.219596 27592 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:34.219717 27592 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:34.219790 27592 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:34.252440 27592 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.70:44045
I20240503 14:12:34.252532 27718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.70:44045 every 8 connection(s)
I20240503 14:12:34.253862 27592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:12:34.257920 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 27592
I20240503 14:12:34.258105 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 14:12:34.261019 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.71:0
--local_ip_for_outbound_sockets=127.22.242.71
--webserver_interface=127.22.242.71
--webserver_port=0
--tserver_master_addrs=127.22.242.126:45965
--builtin_ntp_servers=127.22.242.84:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:34.267563 27719 heartbeater.cc:347] Connected to a master server at 127.22.242.126:45965
I20240503 14:12:34.267912 27719 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:34.269055 27719 heartbeater.cc:511] Master 127.22.242.126:45965 requested a full tablet report, sending...
I20240503 14:12:34.270452 26882 ts_manager.cc:194] Registered new tserver with Master: 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70:44045)
I20240503 14:12:34.271186 26882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.70:59651
I20240503 14:12:34.333274 27459 heartbeater.cc:503] Master 127.22.242.126:45965 was elected leader, sending a full tablet report...
W20240503 14:12:34.602298 27722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:34.602636 27722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:34.602739 27722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:34.610410 27722 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:34.610558 27722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:34.610678 27722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.71
I20240503 14:12:34.619815 27722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.242.71
--webserver_port=0
--tserver_master_addrs=127.22.242.126:45965
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.71
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:34.621207 27722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:34.622949 27722 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:34.634718 27728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:34.634672 27727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:34.634688 27730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:34.635519 27722 server_base.cc:1034] running on GCE node
I20240503 14:12:34.636202 27722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:34.637194 27722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:34.638420 27722 hybrid_clock.cc:648] HybridClock initialized: now 1714745554638371 us; error 38 us; skew 500 ppm
I20240503 14:12:34.638813 27722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:34.641284 27722 webserver.cc:460] Webserver started at http://127.22.242.71:33157/ using document root <none> and password file <none>
I20240503 14:12:34.642203 27722 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:34.642336 27722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:34.642678 27722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:34.645417 27722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0"
format_stamp: "Formatted at 2024-05-03 14:12:34 on dist-test-slave-r3pd"
I20240503 14:12:34.646199 27722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0"
format_stamp: "Formatted at 2024-05-03 14:12:34 on dist-test-slave-r3pd"
I20240503 14:12:34.651383 27722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20240503 14:12:34.654812 27736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:34.656215 27722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240503 14:12:34.656378 27722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0"
format_stamp: "Formatted at 2024-05-03 14:12:34 on dist-test-slave-r3pd"
I20240503 14:12:34.656579 27722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:34.682387 27722 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:34.683288 27722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:34.683563 27722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:34.684630 27722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:34.686194 27722 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:34.686296 27722 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:34.686416 27722 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:34.686491 27722 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:34.717165 27722 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.71:42765
I20240503 14:12:34.717242 27848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.71:42765 every 8 connection(s)
I20240503 14:12:34.718492 27722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:12:34.726051 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 27722
I20240503 14:12:34.726246 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 14:12:34.731379 27849 heartbeater.cc:347] Connected to a master server at 127.22.242.126:45965
I20240503 14:12:34.731666 27849 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:34.732484 27849 heartbeater.cc:511] Master 127.22.242.126:45965 requested a full tablet report, sending...
I20240503 14:12:34.733769 26882 ts_manager.cc:194] Registered new tserver with Master: 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71:42765)
I20240503 14:12:34.734519 26882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.71:43613
I20240503 14:12:34.742185 23497 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 14:12:34.757545 26882 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56310:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 14:12:34.785322 27134 tablet_service.cc:1467] Processing CreateTablet for tablet 09ca4d63eab743b3977a47bcf120b3cf (DEFAULT_TABLE table=TestTable [id=fd924edbfa2840c2bca0f99e7e0778a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:34.785322 27654 tablet_service.cc:1467] Processing CreateTablet for tablet 09ca4d63eab743b3977a47bcf120b3cf (DEFAULT_TABLE table=TestTable [id=fd924edbfa2840c2bca0f99e7e0778a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:34.786892 27134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ca4d63eab743b3977a47bcf120b3cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:34.786911 27654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ca4d63eab743b3977a47bcf120b3cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:34.800863 27865 tablet_bootstrap.cc:492] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617: Bootstrap starting.
I20240503 14:12:34.800863 27866 tablet_bootstrap.cc:492] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3: Bootstrap starting.
I20240503 14:12:34.804754 27866 tablet_bootstrap.cc:654] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:34.804754 27865 tablet_bootstrap.cc:654] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:34.806159 27865 log.cc:824] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:34.806183 27866 log.cc:824] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:34.809190 27866 tablet_bootstrap.cc:492] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3: No bootstrap required, opened a new log
I20240503 14:12:34.809353 27865 tablet_bootstrap.cc:492] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617: No bootstrap required, opened a new log
I20240503 14:12:34.809504 27866 ts_tablet_manager.cc:1367] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240503 14:12:34.809588 27865 ts_tablet_manager.cc:1367] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240503 14:12:34.812734 27589 heartbeater.cc:503] Master 127.22.242.126:45965 was elected leader, sending a full tablet report...
I20240503 14:12:34.818755 27865 raft_consensus.cc:348] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } }
I20240503 14:12:34.818756 27866 raft_consensus.cc:348] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } }
I20240503 14:12:34.819247 27866 raft_consensus.cc:374] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:34.819247 27865 raft_consensus.cc:374] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:34.819397 27866 raft_consensus.cc:729] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89a56aa776934c4582bfbdb0622b0cb3, State: Initialized, Role: FOLLOWER
I20240503 14:12:34.819397 27865 raft_consensus.cc:729] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305daa9a4d874d9db0e5378e5d27b617, State: Initialized, Role: FOLLOWER
I20240503 14:12:34.820219 27866 consensus_queue.cc:260] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } }
I20240503 14:12:34.820219 27865 consensus_queue.cc:260] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } }
I20240503 14:12:34.821504 27719 heartbeater.cc:503] Master 127.22.242.126:45965 was elected leader, sending a full tablet report...
I20240503 14:12:34.822247 27866 ts_tablet_manager.cc:1398] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20240503 14:12:34.823101 27865 ts_tablet_manager.cc:1398] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20240503 14:12:34.840389 27869 raft_consensus.cc:482] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:34.840653 27869 raft_consensus.cc:504] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } }
I20240503 14:12:34.842178 27869 leader_election.cc:288] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:34.848124 27154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ca4d63eab743b3977a47bcf120b3cf" candidate_uuid: "89a56aa776934c4582bfbdb0622b0cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305daa9a4d874d9db0e5378e5d27b617" is_pre_election: true
I20240503 14:12:34.848626 27154 raft_consensus.cc:2454] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89a56aa776934c4582bfbdb0622b0cb3 in term 0.
I20240503 14:12:34.849181 27609 leader_election.cc:302] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, 89a56aa776934c4582bfbdb0622b0cb3; no voters:
I20240503 14:12:34.849547 27869 raft_consensus.cc:2781] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:34.849685 27869 raft_consensus.cc:482] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:34.849778 27869 raft_consensus.cc:3037] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:34.851780 27869 raft_consensus.cc:504] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } }
I20240503 14:12:34.852362 27869 leader_election.cc:288] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [CANDIDATE]: Term 1 election: Requested vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:34.852816 27154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ca4d63eab743b3977a47bcf120b3cf" candidate_uuid: "89a56aa776934c4582bfbdb0622b0cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305daa9a4d874d9db0e5378e5d27b617"
I20240503 14:12:34.852990 27154 raft_consensus.cc:3037] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:34.855173 27154 raft_consensus.cc:2454] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89a56aa776934c4582bfbdb0622b0cb3 in term 1.
I20240503 14:12:34.855540 27609 leader_election.cc:302] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, 89a56aa776934c4582bfbdb0622b0cb3; no voters:
I20240503 14:12:34.855809 27869 raft_consensus.cc:2781] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:34.856238 27869 raft_consensus.cc:686] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 LEADER]: Becoming Leader. State: Replica: 89a56aa776934c4582bfbdb0622b0cb3, State: Running, Role: LEADER
I20240503 14:12:34.856748 27869 consensus_queue.cc:237] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } }
I20240503 14:12:34.860899 26881 catalog_manager.cc:5484] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70). New cstate: current_term: 1 leader_uuid: "89a56aa776934c4582bfbdb0622b0cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:34.878553 23497 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 14:12:34.880518 23497 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 820e18ddcc1e4d53821d43bc32652bc9 to finish bootstrapping
I20240503 14:12:34.889525 23497 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 305daa9a4d874d9db0e5378e5d27b617 to finish bootstrapping
I20240503 14:12:34.896529 23497 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f6c292faf79e49edabdbc7a310a30503 to finish bootstrapping
I20240503 14:12:34.905113 23497 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4165ee79ee03466981e067d998339611 to finish bootstrapping
W20240503 14:12:34.908363 27200 tablet.cc:2369] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:34.914036 23497 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cbb4bf53151a4061a81cf5bacf084516 to finish bootstrapping
I20240503 14:12:34.922400 23497 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 89a56aa776934c4582bfbdb0622b0cb3 to finish bootstrapping
I20240503 14:12:34.929231 23497 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6b69a8c25e2b4de0bb10db8d7bdf83f0 to finish bootstrapping
I20240503 14:12:34.938122 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 27332
I20240503 14:12:34.939154 27454 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:12:35.005826 27720 tablet.cc:2369] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:35.020136 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 27332
I20240503 14:12:35.036353 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 27462
I20240503 14:12:35.037408 27584 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:35.117070 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 27462
I20240503 14:12:35.133272 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 27592
I20240503 14:12:35.134378 27714 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:35.220391 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 27592
I20240503 14:12:35.237289 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 27722
I20240503 14:12:35.238395 27844 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:35.318995 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 27722
I20240503 14:12:36.691253 27884 raft_consensus.cc:482] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:36.691468 27884 raft_consensus.cc:504] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } }
I20240503 14:12:36.693197 27884 leader_election.cc:288] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70:44045)
W20240503 14:12:36.701084 27089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.242.70:44045: connect: Connection refused (error 111)
W20240503 14:12:36.702992 27089 leader_election.cc:334] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70:44045): Network error: Client connection negotiation failed: client connection to 127.22.242.70:44045: connect: Connection refused (error 111)
I20240503 14:12:36.703253 27089 leader_election.cc:302] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617; no voters: 89a56aa776934c4582bfbdb0622b0cb3
I20240503 14:12:36.703711 27884 raft_consensus.cc:2726] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:12:38.228955 27887 raft_consensus.cc:482] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:38.229167 27887 raft_consensus.cc:504] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } }
I20240503 14:12:38.229734 27887 leader_election.cc:288] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70:44045)
W20240503 14:12:38.233176 27089 leader_election.cc:334] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70:44045): Network error: Client connection negotiation failed: client connection to 127.22.242.70:44045: connect: Connection refused (error 111)
I20240503 14:12:38.233309 27089 leader_election.cc:302] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617; no voters: 89a56aa776934c4582bfbdb0622b0cb3
I20240503 14:12:38.233572 27887 raft_consensus.cc:2726] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:12:39.086572 26881 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56310:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:39.098037 27134 tablet_service.cc:1467] Processing CreateTablet for tablet eee65dca344f41528758a1babd90a2ff (DEFAULT_TABLE table=rebalance_test_table_0 [id=eda67e22fc6146289f3862c2470418fb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:39.098670 27134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee65dca344f41528758a1babd90a2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:39.110993 27898 tablet_bootstrap.cc:492] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617: Bootstrap starting.
I20240503 14:12:39.113672 27898 tablet_bootstrap.cc:654] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:39.117239 27002 tablet_service.cc:1467] Processing CreateTablet for tablet 7efea07ad3204b6e868a05e9d80f5b3a (DEFAULT_TABLE table=rebalance_test_table_0 [id=eda67e22fc6146289f3862c2470418fb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:39.117591 27001 tablet_service.cc:1467] Processing CreateTablet for tablet eee65dca344f41528758a1babd90a2ff (DEFAULT_TABLE table=rebalance_test_table_0 [id=eda67e22fc6146289f3862c2470418fb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:39.118937 27003 tablet_service.cc:1467] Processing CreateTablet for tablet 27288fd1dcc8421dab42d3c282f718ae (DEFAULT_TABLE table=rebalance_test_table_0 [id=eda67e22fc6146289f3862c2470418fb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:39.119302 27002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7efea07ad3204b6e868a05e9d80f5b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:39.119932 27003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27288fd1dcc8421dab42d3c282f718ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:39.120250 27001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee65dca344f41528758a1babd90a2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:39.123258 27898 tablet_bootstrap.cc:492] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617: No bootstrap required, opened a new log
I20240503 14:12:39.123451 27898 ts_tablet_manager.cc:1367] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240503 14:12:39.122994 27263 tablet_service.cc:1467] Processing CreateTablet for tablet 7efea07ad3204b6e868a05e9d80f5b3a (DEFAULT_TABLE table=rebalance_test_table_0 [id=eda67e22fc6146289f3862c2470418fb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:39.122800 27264 tablet_service.cc:1467] Processing CreateTablet for tablet 27288fd1dcc8421dab42d3c282f718ae (DEFAULT_TABLE table=rebalance_test_table_0 [id=eda67e22fc6146289f3862c2470418fb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:39.124517 27264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27288fd1dcc8421dab42d3c282f718ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:39.124475 27898 raft_consensus.cc:348] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:39.124730 27898 raft_consensus.cc:374] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:39.124835 27898 raft_consensus.cc:729] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305daa9a4d874d9db0e5378e5d27b617, State: Initialized, Role: FOLLOWER
I20240503 14:12:39.125097 27898 consensus_queue.cc:260] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:39.125501 27263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7efea07ad3204b6e868a05e9d80f5b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:39.134411 27898 ts_tablet_manager.cc:1398] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617: Time spent starting tablet: real 0.011s user 0.003s sys 0.008s
I20240503 14:12:39.142819 27901 tablet_bootstrap.cc:492] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9: Bootstrap starting.
I20240503 14:12:39.143239 27902 tablet_bootstrap.cc:492] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503: Bootstrap starting.
I20240503 14:12:39.146065 27901 tablet_bootstrap.cc:654] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:39.146062 27902 tablet_bootstrap.cc:654] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:39.147153 27902 log.cc:824] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:39.147156 27901 log.cc:824] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:39.150224 27901 tablet_bootstrap.cc:492] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9: No bootstrap required, opened a new log
I20240503 14:12:39.150219 27902 tablet_bootstrap.cc:492] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503: No bootstrap required, opened a new log
I20240503 14:12:39.150498 27902 ts_tablet_manager.cc:1367] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:12:39.150498 27901 ts_tablet_manager.cc:1367] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 14:12:39.159539 27901 raft_consensus.cc:348] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:39.159539 27902 raft_consensus.cc:348] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:39.159998 27902 raft_consensus.cc:374] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:39.159998 27901 raft_consensus.cc:374] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:39.160111 27901 raft_consensus.cc:729] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Initialized, Role: FOLLOWER
I20240503 14:12:39.160116 27902 raft_consensus.cc:729] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c292faf79e49edabdbc7a310a30503, State: Initialized, Role: FOLLOWER
I20240503 14:12:39.160820 27902 consensus_queue.cc:260] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:39.160841 27901 consensus_queue.cc:260] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:39.162746 27902 ts_tablet_manager.cc:1398] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503: Time spent starting tablet: real 0.012s user 0.004s sys 0.009s
I20240503 14:12:39.163136 27902 tablet_bootstrap.cc:492] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503: Bootstrap starting.
I20240503 14:12:39.164644 27901 ts_tablet_manager.cc:1398] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20240503 14:12:39.164836 27902 tablet_bootstrap.cc:654] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:39.165114 27901 tablet_bootstrap.cc:492] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9: Bootstrap starting.
I20240503 14:12:39.166719 27901 tablet_bootstrap.cc:654] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:39.169312 27901 tablet_bootstrap.cc:492] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9: No bootstrap required, opened a new log
I20240503 14:12:39.169447 27901 ts_tablet_manager.cc:1367] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:39.169745 27902 tablet_bootstrap.cc:492] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503: No bootstrap required, opened a new log
I20240503 14:12:39.169868 27902 ts_tablet_manager.cc:1367] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 14:12:39.170219 27901 raft_consensus.cc:348] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:39.170372 27901 raft_consensus.cc:374] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:39.170464 27901 raft_consensus.cc:729] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Initialized, Role: FOLLOWER
I20240503 14:12:39.170596 27902 raft_consensus.cc:348] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:39.170737 27902 raft_consensus.cc:374] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:39.170693 27901 consensus_queue.cc:260] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:39.170825 27902 raft_consensus.cc:729] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c292faf79e49edabdbc7a310a30503, State: Initialized, Role: FOLLOWER
I20240503 14:12:39.171142 27901 ts_tablet_manager.cc:1398] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:39.171058 27902 consensus_queue.cc:260] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:39.171545 27902 ts_tablet_manager.cc:1398] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:39.171835 27901 tablet_bootstrap.cc:492] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9: Bootstrap starting.
I20240503 14:12:39.173393 27901 tablet_bootstrap.cc:654] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:39.175499 27901 tablet_bootstrap.cc:492] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9: No bootstrap required, opened a new log
I20240503 14:12:39.175644 27901 ts_tablet_manager.cc:1367] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:39.176136 27901 raft_consensus.cc:348] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:39.176347 27901 raft_consensus.cc:374] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:39.176421 27901 raft_consensus.cc:729] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Initialized, Role: FOLLOWER
I20240503 14:12:39.176659 27901 consensus_queue.cc:260] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:39.177131 27901 ts_tablet_manager.cc:1398] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 14:12:39.214826 27069 tablet.cc:2369] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:39.236790 27905 raft_consensus.cc:482] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:39.237046 27905 raft_consensus.cc:504] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:39.238559 27905 leader_election.cc:288] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423)
I20240503 14:12:39.244885 27284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efea07ad3204b6e868a05e9d80f5b3a" candidate_uuid: "820e18ddcc1e4d53821d43bc32652bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c292faf79e49edabdbc7a310a30503" is_pre_election: true
I20240503 14:12:39.245409 27284 raft_consensus.cc:2454] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 820e18ddcc1e4d53821d43bc32652bc9 in term 0.
I20240503 14:12:39.246147 26956 leader_election.cc:302] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 820e18ddcc1e4d53821d43bc32652bc9, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:39.246557 27905 raft_consensus.cc:2781] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:39.246691 27905 raft_consensus.cc:482] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:39.246779 27905 raft_consensus.cc:3037] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:39.249861 27905 raft_consensus.cc:504] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:39.250324 27905 leader_election.cc:288] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 election: Requested vote from peers f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423)
I20240503 14:12:39.250833 27284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efea07ad3204b6e868a05e9d80f5b3a" candidate_uuid: "820e18ddcc1e4d53821d43bc32652bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c292faf79e49edabdbc7a310a30503"
I20240503 14:12:39.250998 27284 raft_consensus.cc:3037] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:39.253360 27284 raft_consensus.cc:2454] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 820e18ddcc1e4d53821d43bc32652bc9 in term 1.
I20240503 14:12:39.253747 26956 leader_election.cc:302] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 820e18ddcc1e4d53821d43bc32652bc9, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:39.254081 27905 raft_consensus.cc:2781] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:39.254490 27905 raft_consensus.cc:686] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Running, Role: LEADER
I20240503 14:12:39.254995 27905 consensus_queue.cc:237] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:39.259115 26881 catalog_manager.cc:5484] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:39.335529 27906 raft_consensus.cc:482] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:39.335716 27906 raft_consensus.cc:504] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:39.337162 27906 leader_election.cc:288] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
I20240503 14:12:39.343181 27023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27288fd1dcc8421dab42d3c282f718ae" candidate_uuid: "f6c292faf79e49edabdbc7a310a30503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9" is_pre_election: true
I20240503 14:12:39.343564 27023 raft_consensus.cc:2454] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6c292faf79e49edabdbc7a310a30503 in term 0.
I20240503 14:12:39.344080 27217 leader_election.cc:302] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 820e18ddcc1e4d53821d43bc32652bc9, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:39.344514 27906 raft_consensus.cc:2781] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:39.344647 27906 raft_consensus.cc:482] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:39.344727 27906 raft_consensus.cc:3037] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:39.346730 27906 raft_consensus.cc:504] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:39.347196 27906 leader_election.cc:288] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 election: Requested vote from peers 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
I20240503 14:12:39.347666 27023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27288fd1dcc8421dab42d3c282f718ae" candidate_uuid: "f6c292faf79e49edabdbc7a310a30503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9"
I20240503 14:12:39.347838 27023 raft_consensus.cc:3037] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:39.349828 27023 raft_consensus.cc:2454] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6c292faf79e49edabdbc7a310a30503 in term 1.
I20240503 14:12:39.350231 27217 leader_election.cc:302] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 820e18ddcc1e4d53821d43bc32652bc9, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:39.350497 27906 raft_consensus.cc:2781] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:39.350924 27906 raft_consensus.cc:686] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Becoming Leader. State: Replica: f6c292faf79e49edabdbc7a310a30503, State: Running, Role: LEADER
I20240503 14:12:39.351346 27906 consensus_queue.cc:237] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:39.355768 26881 catalog_manager.cc:5484] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6c292faf79e49edabdbc7a310a30503 (127.22.242.67). New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: HEALTHY } } }
W20240503 14:12:39.372565 27330 tablet.cc:2369] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:39.380559 27905 raft_consensus.cc:482] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:39.380790 27905 raft_consensus.cc:504] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:39.381358 27905 leader_election.cc:288] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:39.386693 27154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee65dca344f41528758a1babd90a2ff" candidate_uuid: "820e18ddcc1e4d53821d43bc32652bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305daa9a4d874d9db0e5378e5d27b617" is_pre_election: true
I20240503 14:12:39.386932 27154 raft_consensus.cc:2454] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 820e18ddcc1e4d53821d43bc32652bc9 in term 0.
I20240503 14:12:39.387394 26958 leader_election.cc:302] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, 820e18ddcc1e4d53821d43bc32652bc9; no voters:
I20240503 14:12:39.387727 27905 raft_consensus.cc:2781] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:39.387814 27905 raft_consensus.cc:482] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:39.387943 27905 raft_consensus.cc:3037] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:39.389825 27905 raft_consensus.cc:504] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:39.390311 27905 leader_election.cc:288] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 election: Requested vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:39.390746 27154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee65dca344f41528758a1babd90a2ff" candidate_uuid: "820e18ddcc1e4d53821d43bc32652bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305daa9a4d874d9db0e5378e5d27b617"
I20240503 14:12:39.390925 27154 raft_consensus.cc:3037] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:39.392933 27154 raft_consensus.cc:2454] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 820e18ddcc1e4d53821d43bc32652bc9 in term 1.
I20240503 14:12:39.393306 26958 leader_election.cc:302] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, 820e18ddcc1e4d53821d43bc32652bc9; no voters:
I20240503 14:12:39.393587 27905 raft_consensus.cc:2781] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:39.393748 27905 raft_consensus.cc:686] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Running, Role: LEADER
I20240503 14:12:39.393999 27905 consensus_queue.cc:237] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:39.396138 26881 catalog_manager.cc:5484] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:39.793241 27906 consensus_queue.cc:1035] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:39.793313 27918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:39.793648 27918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:39.803524 27918 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 14:12:39.841982 27905 consensus_queue.cc:1035] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:39.883119 27911 consensus_queue.cc:1035] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:39.991847 27937 raft_consensus.cc:482] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:39.992110 27937 raft_consensus.cc:504] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } }
I20240503 14:12:39.992915 27937 leader_election.cc:288] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70:44045)
W20240503 14:12:40.017900 27089 leader_election.cc:334] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70:44045): Network error: Client connection negotiation failed: client connection to 127.22.242.70:44045: connect: Connection refused (error 111)
I20240503 14:12:40.018124 27089 leader_election.cc:302] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617; no voters: 89a56aa776934c4582bfbdb0622b0cb3
I20240503 14:12:40.018381 27937 raft_consensus.cc:2726] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
INSERT report
rows total: 2000
time total: 225.695 ms
time per row: 0.112848 ms
I20240503 14:12:40.180555 26873 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56310:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:40.188537 27264 tablet_service.cc:1467] Processing CreateTablet for tablet f2db6b6ac8bb4fffb18e0b65dea27b5c (DEFAULT_TABLE table=rebalance_test_table_1 [id=34a5ee8103e54b178f47b78610c6a027]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:40.188707 27134 tablet_service.cc:1467] Processing CreateTablet for tablet f2db6b6ac8bb4fffb18e0b65dea27b5c (DEFAULT_TABLE table=rebalance_test_table_1 [id=34a5ee8103e54b178f47b78610c6a027]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:40.189043 27264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2db6b6ac8bb4fffb18e0b65dea27b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:40.189141 27134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2db6b6ac8bb4fffb18e0b65dea27b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:40.189720 27003 tablet_service.cc:1467] Processing CreateTablet for tablet c6d0c7755fb8449ea077c3786dc619fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=34a5ee8103e54b178f47b78610c6a027]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:40.190107 27003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d0c7755fb8449ea077c3786dc619fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:40.190145 27263 tablet_service.cc:1467] Processing CreateTablet for tablet efa780a04e854838a933be5934cd1578 (DEFAULT_TABLE table=rebalance_test_table_1 [id=34a5ee8103e54b178f47b78610c6a027]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:40.190507 27263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa780a04e854838a933be5934cd1578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:40.189043 27133 tablet_service.cc:1467] Processing CreateTablet for tablet efa780a04e854838a933be5934cd1578 (DEFAULT_TABLE table=rebalance_test_table_1 [id=34a5ee8103e54b178f47b78610c6a027]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:40.192967 27133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa780a04e854838a933be5934cd1578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:40.196823 27132 tablet_service.cc:1467] Processing CreateTablet for tablet c6d0c7755fb8449ea077c3786dc619fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=34a5ee8103e54b178f47b78610c6a027]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:40.197532 27132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d0c7755fb8449ea077c3786dc619fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:40.208417 27975 tablet_bootstrap.cc:492] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617: Bootstrap starting.
I20240503 14:12:40.209096 27976 tablet_bootstrap.cc:492] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9: Bootstrap starting.
I20240503 14:12:40.210609 27976 tablet_bootstrap.cc:654] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:40.211077 27975 tablet_bootstrap.cc:654] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:40.212448 27974 tablet_bootstrap.cc:492] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503: Bootstrap starting.
I20240503 14:12:40.213459 27976 tablet_bootstrap.cc:492] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9: No bootstrap required, opened a new log
I20240503 14:12:40.213459 27975 tablet_bootstrap.cc:492] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617: No bootstrap required, opened a new log
I20240503 14:12:40.213618 27975 ts_tablet_manager.cc:1367] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 14:12:40.213618 27976 ts_tablet_manager.cc:1367] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:12:40.214044 27974 tablet_bootstrap.cc:654] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:40.214390 27975 raft_consensus.cc:348] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:40.214397 27976 raft_consensus.cc:348] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:40.214607 27975 raft_consensus.cc:374] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:40.214607 27976 raft_consensus.cc:374] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:40.214704 27976 raft_consensus.cc:729] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Initialized, Role: FOLLOWER
I20240503 14:12:40.214754 27975 raft_consensus.cc:729] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305daa9a4d874d9db0e5378e5d27b617, State: Initialized, Role: FOLLOWER
I20240503 14:12:40.214946 27976 consensus_queue.cc:260] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:40.215031 27975 consensus_queue.cc:260] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:40.215553 27976 ts_tablet_manager.cc:1398] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:40.216403 27975 ts_tablet_manager.cc:1398] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:12:40.216701 27975 tablet_bootstrap.cc:492] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617: Bootstrap starting.
I20240503 14:12:40.217137 27974 tablet_bootstrap.cc:492] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503: No bootstrap required, opened a new log
I20240503 14:12:40.217356 27974 ts_tablet_manager.cc:1367] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:12:40.218969 27974 raft_consensus.cc:348] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:40.219142 27974 raft_consensus.cc:374] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:40.219265 27974 raft_consensus.cc:729] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c292faf79e49edabdbc7a310a30503, State: Initialized, Role: FOLLOWER
I20240503 14:12:40.219415 27975 tablet_bootstrap.cc:654] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:40.219487 27974 consensus_queue.cc:260] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:40.221122 27974 ts_tablet_manager.cc:1398] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 14:12:40.221383 27974 tablet_bootstrap.cc:492] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503: Bootstrap starting.
I20240503 14:12:40.221940 27975 tablet_bootstrap.cc:492] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617: No bootstrap required, opened a new log
I20240503 14:12:40.222164 27975 ts_tablet_manager.cc:1367] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:12:40.222846 27975 raft_consensus.cc:348] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:40.223059 27975 raft_consensus.cc:374] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:40.223146 27975 raft_consensus.cc:729] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305daa9a4d874d9db0e5378e5d27b617, State: Initialized, Role: FOLLOWER
I20240503 14:12:40.223407 27974 tablet_bootstrap.cc:654] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:40.223408 27975 consensus_queue.cc:260] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:40.223966 27975 ts_tablet_manager.cc:1398] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:40.224292 27975 tablet_bootstrap.cc:492] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617: Bootstrap starting.
I20240503 14:12:40.225823 27975 tablet_bootstrap.cc:654] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:40.225860 27974 tablet_bootstrap.cc:492] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503: No bootstrap required, opened a new log
I20240503 14:12:40.226006 27974 ts_tablet_manager.cc:1367] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:12:40.226887 27974 raft_consensus.cc:348] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:40.227068 27974 raft_consensus.cc:374] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:40.227174 27974 raft_consensus.cc:729] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c292faf79e49edabdbc7a310a30503, State: Initialized, Role: FOLLOWER
I20240503 14:12:40.227432 27974 consensus_queue.cc:260] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:40.228005 27974 ts_tablet_manager.cc:1398] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:40.229279 27975 tablet_bootstrap.cc:492] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617: No bootstrap required, opened a new log
I20240503 14:12:40.229403 27975 ts_tablet_manager.cc:1367] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240503 14:12:40.230019 27975 raft_consensus.cc:348] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:40.230186 27975 raft_consensus.cc:374] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:40.230279 27975 raft_consensus.cc:729] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305daa9a4d874d9db0e5378e5d27b617, State: Initialized, Role: FOLLOWER
I20240503 14:12:40.230509 27975 consensus_queue.cc:260] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:40.230950 27975 ts_tablet_manager.cc:1398] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:12:40.302901 27906 raft_consensus.cc:482] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:40.303115 27906 raft_consensus.cc:504] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:40.303820 27906 leader_election.cc:288] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:40.310081 27154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa780a04e854838a933be5934cd1578" candidate_uuid: "f6c292faf79e49edabdbc7a310a30503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305daa9a4d874d9db0e5378e5d27b617" is_pre_election: true
I20240503 14:12:40.310335 27154 raft_consensus.cc:2454] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6c292faf79e49edabdbc7a310a30503 in term 0.
I20240503 14:12:40.310760 27219 leader_election.cc:302] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:40.311038 27906 raft_consensus.cc:2781] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:40.311189 27906 raft_consensus.cc:482] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:40.311260 27906 raft_consensus.cc:3037] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:40.313135 27906 raft_consensus.cc:504] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:40.313599 27906 leader_election.cc:288] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 election: Requested vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:40.314162 27154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa780a04e854838a933be5934cd1578" candidate_uuid: "f6c292faf79e49edabdbc7a310a30503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305daa9a4d874d9db0e5378e5d27b617"
I20240503 14:12:40.314343 27154 raft_consensus.cc:3037] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:40.316272 27154 raft_consensus.cc:2454] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6c292faf79e49edabdbc7a310a30503 in term 1.
I20240503 14:12:40.316741 27219 leader_election.cc:302] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:40.317056 27906 raft_consensus.cc:2781] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:40.317200 27906 raft_consensus.cc:686] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Becoming Leader. State: Replica: f6c292faf79e49edabdbc7a310a30503, State: Running, Role: LEADER
I20240503 14:12:40.317445 27906 consensus_queue.cc:237] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:40.319990 26873 catalog_manager.cc:5484] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6c292faf79e49edabdbc7a310a30503 (127.22.242.67). New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:40.400457 27937 raft_consensus.cc:482] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:40.400657 27937 raft_consensus.cc:504] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:40.401350 27937 leader_election.cc:288] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423)
I20240503 14:12:40.408464 27284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2db6b6ac8bb4fffb18e0b65dea27b5c" candidate_uuid: "305daa9a4d874d9db0e5378e5d27b617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c292faf79e49edabdbc7a310a30503" is_pre_election: true
I20240503 14:12:40.408758 27284 raft_consensus.cc:2454] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 305daa9a4d874d9db0e5378e5d27b617 in term 0.
I20240503 14:12:40.409302 27087 leader_election.cc:302] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:40.409623 27937 raft_consensus.cc:2781] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:40.409760 27937 raft_consensus.cc:482] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:40.409854 27937 raft_consensus.cc:3037] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:40.412984 27937 raft_consensus.cc:504] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:40.413637 27937 leader_election.cc:288] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 1 election: Requested vote from peers f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423)
I20240503 14:12:40.414254 27284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2db6b6ac8bb4fffb18e0b65dea27b5c" candidate_uuid: "305daa9a4d874d9db0e5378e5d27b617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c292faf79e49edabdbc7a310a30503"
I20240503 14:12:40.414485 27284 raft_consensus.cc:3037] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:40.417363 27284 raft_consensus.cc:2454] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 305daa9a4d874d9db0e5378e5d27b617 in term 1.
I20240503 14:12:40.417898 27087 leader_election.cc:302] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:40.418361 27937 raft_consensus.cc:2781] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:40.418843 27937 raft_consensus.cc:686] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [term 1 LEADER]: Becoming Leader. State: Replica: 305daa9a4d874d9db0e5378e5d27b617, State: Running, Role: LEADER
I20240503 14:12:40.419665 27937 consensus_queue.cc:237] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:40.423501 26873 catalog_manager.cc:5484] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: term changed from 0 to 1, leader changed from <none> to 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "305daa9a4d874d9db0e5378e5d27b617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:40.483096 27966 raft_consensus.cc:482] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:40.483299 27966 raft_consensus.cc:504] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:40.483803 27966 leader_election.cc:288] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:40.484463 27154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d0c7755fb8449ea077c3786dc619fc" candidate_uuid: "820e18ddcc1e4d53821d43bc32652bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305daa9a4d874d9db0e5378e5d27b617" is_pre_election: true
I20240503 14:12:40.484680 27154 raft_consensus.cc:2454] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 820e18ddcc1e4d53821d43bc32652bc9 in term 0.
I20240503 14:12:40.485085 26958 leader_election.cc:302] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, 820e18ddcc1e4d53821d43bc32652bc9; no voters:
I20240503 14:12:40.485376 27966 raft_consensus.cc:2781] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:40.485489 27966 raft_consensus.cc:482] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:40.485580 27966 raft_consensus.cc:3037] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:40.487998 27966 raft_consensus.cc:504] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:40.488476 27966 leader_election.cc:288] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 election: Requested vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:40.488929 27154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d0c7755fb8449ea077c3786dc619fc" candidate_uuid: "820e18ddcc1e4d53821d43bc32652bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305daa9a4d874d9db0e5378e5d27b617"
I20240503 14:12:40.489140 27154 raft_consensus.cc:3037] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:40.490919 27154 raft_consensus.cc:2454] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 820e18ddcc1e4d53821d43bc32652bc9 in term 1.
I20240503 14:12:40.491329 26958 leader_election.cc:302] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, 820e18ddcc1e4d53821d43bc32652bc9; no voters:
I20240503 14:12:40.491638 27966 raft_consensus.cc:2781] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:40.491770 27966 raft_consensus.cc:686] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Running, Role: LEADER
I20240503 14:12:40.492028 27966 consensus_queue.cc:237] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:40.494442 26873 catalog_manager.cc:5484] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:40.805203 27906 consensus_queue.cc:1035] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:40.834136 27989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:40.834484 27989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:40.841962 27989 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 14:12:40.976146 27966 consensus_queue.cc:1035] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:40.987990 27284 raft_consensus.cc:1261] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:40.989348 27937 consensus_queue.cc:1035] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:41.057870 27991 mvcc.cc:205] Tried to move back new op lower bound from 7023597817794392064 to 7023597815482839040. Current Snapshot: MvccSnapshot[applied={T|T < 7023597817794392064}]
I20240503 14:12:41.112214 28036 mvcc.cc:205] Tried to move back new op lower bound from 7023597818192969728 to 7023597815776768000. Current Snapshot: MvccSnapshot[applied={T|T < 7023597818192969728}]
INSERT report
rows total: 2000
time total: 232.896 ms
time per row: 0.116448 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:12:42.144002 26876 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56310:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:42.153016 27003 tablet_service.cc:1467] Processing CreateTablet for tablet 6385b8e7ad5849c9a17136f0967c516f (DEFAULT_TABLE table=rebalance_test_table_2 [id=8c704c1747ee4a41b347496dc856e7bf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:42.153168 27264 tablet_service.cc:1467] Processing CreateTablet for tablet 6385b8e7ad5849c9a17136f0967c516f (DEFAULT_TABLE table=rebalance_test_table_2 [id=8c704c1747ee4a41b347496dc856e7bf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:42.153515 27003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6385b8e7ad5849c9a17136f0967c516f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:42.153625 27264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6385b8e7ad5849c9a17136f0967c516f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:42.154105 27132 tablet_service.cc:1467] Processing CreateTablet for tablet 927839713c994a9d9ad9f071a6943b0d (DEFAULT_TABLE table=rebalance_test_table_2 [id=8c704c1747ee4a41b347496dc856e7bf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:42.154224 27263 tablet_service.cc:1467] Processing CreateTablet for tablet ab63a1d4c9764427a141feb6614fe18f (DEFAULT_TABLE table=rebalance_test_table_2 [id=8c704c1747ee4a41b347496dc856e7bf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:42.154484 27132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927839713c994a9d9ad9f071a6943b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:42.154569 27263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab63a1d4c9764427a141feb6614fe18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:42.160235 27001 tablet_service.cc:1467] Processing CreateTablet for tablet ab63a1d4c9764427a141feb6614fe18f (DEFAULT_TABLE table=rebalance_test_table_2 [id=8c704c1747ee4a41b347496dc856e7bf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:42.160260 27002 tablet_service.cc:1467] Processing CreateTablet for tablet 927839713c994a9d9ad9f071a6943b0d (DEFAULT_TABLE table=rebalance_test_table_2 [id=8c704c1747ee4a41b347496dc856e7bf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:42.160696 27002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927839713c994a9d9ad9f071a6943b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:42.164449 27001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab63a1d4c9764427a141feb6614fe18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:42.164682 28050 tablet_bootstrap.cc:492] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9: Bootstrap starting.
I20240503 14:12:42.166316 28050 tablet_bootstrap.cc:654] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:42.170105 28052 tablet_bootstrap.cc:492] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617: Bootstrap starting.
I20240503 14:12:42.172959 28052 tablet_bootstrap.cc:654] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:42.173385 28050 tablet_bootstrap.cc:492] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9: No bootstrap required, opened a new log
I20240503 14:12:42.173589 28050 ts_tablet_manager.cc:1367] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240503 14:12:42.174427 28050 raft_consensus.cc:348] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:42.174643 28050 raft_consensus.cc:374] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:42.174785 28050 raft_consensus.cc:729] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Initialized, Role: FOLLOWER
I20240503 14:12:42.174846 28051 tablet_bootstrap.cc:492] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503: Bootstrap starting.
I20240503 14:12:42.175031 28050 consensus_queue.cc:260] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:42.175680 28050 ts_tablet_manager.cc:1398] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:42.175987 28050 tablet_bootstrap.cc:492] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9: Bootstrap starting.
I20240503 14:12:42.176486 28051 tablet_bootstrap.cc:654] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:42.179976 28050 tablet_bootstrap.cc:654] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:42.182467 28050 tablet_bootstrap.cc:492] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9: No bootstrap required, opened a new log
I20240503 14:12:42.182638 28050 ts_tablet_manager.cc:1367] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240503 14:12:42.182835 28051 tablet_bootstrap.cc:492] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503: No bootstrap required, opened a new log
I20240503 14:12:42.182989 28051 ts_tablet_manager.cc:1367] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 14:12:42.183789 28051 raft_consensus.cc:348] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:42.183913 28050 raft_consensus.cc:348] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:42.184015 28051 raft_consensus.cc:374] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:42.184108 28050 raft_consensus.cc:374] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:42.184108 28051 raft_consensus.cc:729] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c292faf79e49edabdbc7a310a30503, State: Initialized, Role: FOLLOWER
I20240503 14:12:42.184206 28050 raft_consensus.cc:729] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Initialized, Role: FOLLOWER
I20240503 14:12:42.184373 28051 consensus_queue.cc:260] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:42.184496 28050 consensus_queue.cc:260] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:42.184916 28051 ts_tablet_manager.cc:1398] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:42.185017 28052 tablet_bootstrap.cc:492] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617: No bootstrap required, opened a new log
I20240503 14:12:42.185173 28052 ts_tablet_manager.cc:1367] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20240503 14:12:42.185259 28051 tablet_bootstrap.cc:492] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503: Bootstrap starting.
I20240503 14:12:42.186280 28052 raft_consensus.cc:348] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:42.186514 28052 raft_consensus.cc:374] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:42.186630 28052 raft_consensus.cc:729] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305daa9a4d874d9db0e5378e5d27b617, State: Initialized, Role: FOLLOWER
I20240503 14:12:42.186839 28051 tablet_bootstrap.cc:654] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:42.186893 28052 consensus_queue.cc:260] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:42.187445 28052 ts_tablet_manager.cc:1398] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:42.188184 28050 ts_tablet_manager.cc:1398] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:12:42.188493 28050 tablet_bootstrap.cc:492] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9: Bootstrap starting.
I20240503 14:12:42.189992 28050 tablet_bootstrap.cc:654] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:42.191432 28051 tablet_bootstrap.cc:492] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503: No bootstrap required, opened a new log
I20240503 14:12:42.191582 28051 ts_tablet_manager.cc:1367] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:12:42.192122 28051 raft_consensus.cc:348] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:42.192504 28051 raft_consensus.cc:374] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:42.192600 28051 raft_consensus.cc:729] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c292faf79e49edabdbc7a310a30503, State: Initialized, Role: FOLLOWER
I20240503 14:12:42.192845 28051 consensus_queue.cc:260] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:42.193228 28050 tablet_bootstrap.cc:492] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9: No bootstrap required, opened a new log
I20240503 14:12:42.193367 28050 ts_tablet_manager.cc:1367] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:42.193454 28051 ts_tablet_manager.cc:1398] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:42.194201 28050 raft_consensus.cc:348] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:42.194365 28050 raft_consensus.cc:374] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:42.194447 28050 raft_consensus.cc:729] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Initialized, Role: FOLLOWER
I20240503 14:12:42.194681 28050 consensus_queue.cc:260] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:42.195209 28050 ts_tablet_manager.cc:1398] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:42.273661 28038 raft_consensus.cc:482] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:42.273859 28038 raft_consensus.cc:504] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:42.274371 28038 leader_election.cc:288] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423)
I20240503 14:12:42.275024 27284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6385b8e7ad5849c9a17136f0967c516f" candidate_uuid: "820e18ddcc1e4d53821d43bc32652bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c292faf79e49edabdbc7a310a30503" is_pre_election: true
I20240503 14:12:42.275276 27284 raft_consensus.cc:2454] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 820e18ddcc1e4d53821d43bc32652bc9 in term 0.
I20240503 14:12:42.275753 26956 leader_election.cc:302] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 820e18ddcc1e4d53821d43bc32652bc9, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:42.276036 28038 raft_consensus.cc:2781] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:42.276134 28038 raft_consensus.cc:482] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:42.276244 28038 raft_consensus.cc:3037] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:42.278117 28038 raft_consensus.cc:504] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:42.278548 28038 leader_election.cc:288] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 election: Requested vote from peers f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423)
I20240503 14:12:42.279079 27284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6385b8e7ad5849c9a17136f0967c516f" candidate_uuid: "820e18ddcc1e4d53821d43bc32652bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c292faf79e49edabdbc7a310a30503"
I20240503 14:12:42.279296 27284 raft_consensus.cc:3037] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:42.281177 27284 raft_consensus.cc:2454] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 820e18ddcc1e4d53821d43bc32652bc9 in term 1.
I20240503 14:12:42.281570 26956 leader_election.cc:302] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 820e18ddcc1e4d53821d43bc32652bc9, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:42.281888 28038 raft_consensus.cc:2781] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:42.282253 28038 raft_consensus.cc:686] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Running, Role: LEADER
I20240503 14:12:42.282538 28038 consensus_queue.cc:237] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:42.285132 26874 catalog_manager.cc:5484] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:42.295186 28038 raft_consensus.cc:482] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:42.295433 28038 raft_consensus.cc:504] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:42.295964 28038 leader_election.cc:288] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:42.296623 27154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927839713c994a9d9ad9f071a6943b0d" candidate_uuid: "820e18ddcc1e4d53821d43bc32652bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305daa9a4d874d9db0e5378e5d27b617" is_pre_election: true
I20240503 14:12:42.296826 27154 raft_consensus.cc:2454] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 820e18ddcc1e4d53821d43bc32652bc9 in term 0.
I20240503 14:12:42.297214 26958 leader_election.cc:302] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, 820e18ddcc1e4d53821d43bc32652bc9; no voters:
I20240503 14:12:42.297494 28038 raft_consensus.cc:2781] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:42.297582 28038 raft_consensus.cc:482] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:42.297667 28038 raft_consensus.cc:3037] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:42.299479 28038 raft_consensus.cc:504] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:42.299906 28038 leader_election.cc:288] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 election: Requested vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:42.300398 27154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927839713c994a9d9ad9f071a6943b0d" candidate_uuid: "820e18ddcc1e4d53821d43bc32652bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305daa9a4d874d9db0e5378e5d27b617"
I20240503 14:12:42.300575 27154 raft_consensus.cc:3037] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:42.302402 27154 raft_consensus.cc:2454] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 820e18ddcc1e4d53821d43bc32652bc9 in term 1.
I20240503 14:12:42.302783 26958 leader_election.cc:302] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, 820e18ddcc1e4d53821d43bc32652bc9; no voters:
I20240503 14:12:42.303061 28038 raft_consensus.cc:2781] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:42.303233 28038 raft_consensus.cc:686] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Running, Role: LEADER
I20240503 14:12:42.303496 28038 consensus_queue.cc:237] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:42.305958 26874 catalog_manager.cc:5484] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:42.497342 27914 raft_consensus.cc:482] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:42.497546 27914 raft_consensus.cc:504] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:42.498265 27914 leader_election.cc:288] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
I20240503 14:12:42.498903 27022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab63a1d4c9764427a141feb6614fe18f" candidate_uuid: "f6c292faf79e49edabdbc7a310a30503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9" is_pre_election: true
I20240503 14:12:42.499125 27022 raft_consensus.cc:2454] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6c292faf79e49edabdbc7a310a30503 in term 0.
I20240503 14:12:42.499637 27217 leader_election.cc:302] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 820e18ddcc1e4d53821d43bc32652bc9, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:42.499950 27914 raft_consensus.cc:2781] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:42.500118 27914 raft_consensus.cc:482] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:42.500254 27914 raft_consensus.cc:3037] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:42.502140 27914 raft_consensus.cc:504] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:42.502617 27914 leader_election.cc:288] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 election: Requested vote from peers 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
I20240503 14:12:42.503074 27022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab63a1d4c9764427a141feb6614fe18f" candidate_uuid: "f6c292faf79e49edabdbc7a310a30503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9"
I20240503 14:12:42.503258 27022 raft_consensus.cc:3037] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:42.505295 27022 raft_consensus.cc:2454] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6c292faf79e49edabdbc7a310a30503 in term 1.
I20240503 14:12:42.505728 27217 leader_election.cc:302] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 820e18ddcc1e4d53821d43bc32652bc9, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:42.506053 27914 raft_consensus.cc:2781] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:42.506443 27914 raft_consensus.cc:686] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Becoming Leader. State: Replica: f6c292faf79e49edabdbc7a310a30503, State: Running, Role: LEADER
I20240503 14:12:42.506750 27914 consensus_queue.cc:237] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:42.509395 26874 catalog_manager.cc:5484] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6c292faf79e49edabdbc7a310a30503 (127.22.242.67). New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:42.804342 28038 consensus_queue.cc:1035] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:42.806349 28065 consensus_queue.cc:1035] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:42.867465 28064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:42.867779 28064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:42.875243 28064 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 14:12:42.983510 27914 consensus_queue.cc:1035] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:43.110219 28045 raft_consensus.cc:482] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:43.110508 28045 raft_consensus.cc:504] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } }
I20240503 14:12:43.111357 28045 leader_election.cc:288] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70:44045)
W20240503 14:12:43.129549 27089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.242.70:44045: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240503 14:12:43.135550 27089 leader_election.cc:334] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70:44045): Network error: Client connection negotiation failed: client connection to 127.22.242.70:44045: connect: Connection refused (error 111)
I20240503 14:12:43.135749 27089 leader_election.cc:302] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617; no voters: 89a56aa776934c4582bfbdb0622b0cb3
I20240503 14:12:43.136056 28045 raft_consensus.cc:2726] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
INSERT report
rows total: 2000
time total: 222.378 ms
time per row: 0.111189 ms
I20240503 14:12:43.244360 26882 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56310:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:43.253190 27132 tablet_service.cc:1467] Processing CreateTablet for tablet f24cc60aba81429ca7275571faf77cb9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=da6fa437d89c4c1184580d93ab404751]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:43.253384 27264 tablet_service.cc:1467] Processing CreateTablet for tablet f24cc60aba81429ca7275571faf77cb9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=da6fa437d89c4c1184580d93ab404751]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:43.253716 27132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f24cc60aba81429ca7275571faf77cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:43.253845 27264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f24cc60aba81429ca7275571faf77cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:43.254611 27001 tablet_service.cc:1467] Processing CreateTablet for tablet c47ed0ca358f4370b61529f8ffba2503 (DEFAULT_TABLE table=rebalance_test_table_3 [id=da6fa437d89c4c1184580d93ab404751]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:43.254684 27263 tablet_service.cc:1467] Processing CreateTablet for tablet 4cca566ca06643ffbc72f02be6d544b4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=da6fa437d89c4c1184580d93ab404751]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:43.255010 27001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c47ed0ca358f4370b61529f8ffba2503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:43.255066 27263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cca566ca06643ffbc72f02be6d544b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:43.260624 27134 tablet_service.cc:1467] Processing CreateTablet for tablet c47ed0ca358f4370b61529f8ffba2503 (DEFAULT_TABLE table=rebalance_test_table_3 [id=da6fa437d89c4c1184580d93ab404751]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:43.260814 27133 tablet_service.cc:1467] Processing CreateTablet for tablet 4cca566ca06643ffbc72f02be6d544b4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=da6fa437d89c4c1184580d93ab404751]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:43.261094 27134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c47ed0ca358f4370b61529f8ffba2503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:43.261332 27133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cca566ca06643ffbc72f02be6d544b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:43.266077 28122 tablet_bootstrap.cc:492] T c47ed0ca358f4370b61529f8ffba2503 P 820e18ddcc1e4d53821d43bc32652bc9: Bootstrap starting.
I20240503 14:12:43.266870 28121 tablet_bootstrap.cc:492] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503: Bootstrap starting.
I20240503 14:12:43.268769 28121 tablet_bootstrap.cc:654] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:43.268828 28122 tablet_bootstrap.cc:654] T c47ed0ca358f4370b61529f8ffba2503 P 820e18ddcc1e4d53821d43bc32652bc9: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:43.269729 28120 tablet_bootstrap.cc:492] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617: Bootstrap starting.
I20240503 14:12:43.271284 28120 tablet_bootstrap.cc:654] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:43.273209 28120 tablet_bootstrap.cc:492] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617: No bootstrap required, opened a new log
I20240503 14:12:43.273341 28120 ts_tablet_manager.cc:1367] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:43.273917 28120 raft_consensus.cc:348] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:43.274214 28120 raft_consensus.cc:374] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:43.274309 28120 raft_consensus.cc:729] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305daa9a4d874d9db0e5378e5d27b617, State: Initialized, Role: FOLLOWER
I20240503 14:12:43.274528 28120 consensus_queue.cc:260] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:43.275884 28120 ts_tablet_manager.cc:1398] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:43.276124 28120 tablet_bootstrap.cc:492] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617: Bootstrap starting.
I20240503 14:12:43.277293 28120 tablet_bootstrap.cc:654] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:43.279104 28121 tablet_bootstrap.cc:492] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503: No bootstrap required, opened a new log
I20240503 14:12:43.279265 28121 ts_tablet_manager.cc:1367] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20240503 14:12:43.279271 28122 tablet_bootstrap.cc:492] T c47ed0ca358f4370b61529f8ffba2503 P 820e18ddcc1e4d53821d43bc32652bc9: No bootstrap required, opened a new log
I20240503 14:12:43.279426 28122 ts_tablet_manager.cc:1367] T c47ed0ca358f4370b61529f8ffba2503 P 820e18ddcc1e4d53821d43bc32652bc9: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20240503 14:12:43.280143 28121 raft_consensus.cc:348] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:43.280251 28122 raft_consensus.cc:348] T c47ed0ca358f4370b61529f8ffba2503 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:43.280527 28122 raft_consensus.cc:374] T c47ed0ca358f4370b61529f8ffba2503 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:43.280519 28121 raft_consensus.cc:374] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:43.280663 28122 raft_consensus.cc:729] T c47ed0ca358f4370b61529f8ffba2503 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Initialized, Role: FOLLOWER
I20240503 14:12:43.280886 28121 raft_consensus.cc:729] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c292faf79e49edabdbc7a310a30503, State: Initialized, Role: FOLLOWER
I20240503 14:12:43.280969 28122 consensus_queue.cc:260] T c47ed0ca358f4370b61529f8ffba2503 P 820e18ddcc1e4d53821d43bc32652bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:43.281127 28120 tablet_bootstrap.cc:492] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617: No bootstrap required, opened a new log
I20240503 14:12:43.281287 28120 ts_tablet_manager.cc:1367] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:12:43.281574 28122 ts_tablet_manager.cc:1398] T c47ed0ca358f4370b61529f8ffba2503 P 820e18ddcc1e4d53821d43bc32652bc9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:43.282059 28120 raft_consensus.cc:348] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:43.282058 28121 consensus_queue.cc:260] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:43.282318 28120 raft_consensus.cc:374] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:43.282438 28120 raft_consensus.cc:729] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305daa9a4d874d9db0e5378e5d27b617, State: Initialized, Role: FOLLOWER
I20240503 14:12:43.282660 28121 ts_tablet_manager.cc:1398] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:12:43.282728 28120 consensus_queue.cc:260] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:43.283136 28121 tablet_bootstrap.cc:492] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503: Bootstrap starting.
I20240503 14:12:43.283277 28120 ts_tablet_manager.cc:1398] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:12:43.283555 28120 tablet_bootstrap.cc:492] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617: Bootstrap starting.
I20240503 14:12:43.285751 28121 tablet_bootstrap.cc:654] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:43.286207 28120 tablet_bootstrap.cc:654] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:43.288420 28120 tablet_bootstrap.cc:492] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617: No bootstrap required, opened a new log
I20240503 14:12:43.288569 28120 ts_tablet_manager.cc:1367] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:12:43.288740 28121 tablet_bootstrap.cc:492] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503: No bootstrap required, opened a new log
I20240503 14:12:43.288895 28121 ts_tablet_manager.cc:1367] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240503 14:12:43.289328 28120 raft_consensus.cc:348] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:43.289563 28120 raft_consensus.cc:374] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:43.289683 28120 raft_consensus.cc:729] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305daa9a4d874d9db0e5378e5d27b617, State: Initialized, Role: FOLLOWER
I20240503 14:12:43.289626 28121 raft_consensus.cc:348] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:43.289829 28121 raft_consensus.cc:374] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:43.289903 28121 raft_consensus.cc:729] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c292faf79e49edabdbc7a310a30503, State: Initialized, Role: FOLLOWER
I20240503 14:12:43.289924 28120 consensus_queue.cc:260] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:43.290144 28121 consensus_queue.cc:260] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:43.290436 28120 ts_tablet_manager.cc:1398] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:43.290639 28121 ts_tablet_manager.cc:1398] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:43.291049 28045 raft_consensus.cc:482] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:43.291177 28045 raft_consensus.cc:504] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:43.291656 28045 leader_election.cc:288] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423)
I20240503 14:12:43.292740 27284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f24cc60aba81429ca7275571faf77cb9" candidate_uuid: "305daa9a4d874d9db0e5378e5d27b617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c292faf79e49edabdbc7a310a30503" is_pre_election: true
I20240503 14:12:43.292940 27284 raft_consensus.cc:2454] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 305daa9a4d874d9db0e5378e5d27b617 in term 0.
I20240503 14:12:43.293393 27087 leader_election.cc:302] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:43.293852 28045 raft_consensus.cc:2781] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:43.293975 28045 raft_consensus.cc:482] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:43.294075 28045 raft_consensus.cc:3037] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:43.296247 28045 raft_consensus.cc:504] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:43.296674 28045 leader_election.cc:288] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 1 election: Requested vote from peers f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423)
I20240503 14:12:43.297183 27284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f24cc60aba81429ca7275571faf77cb9" candidate_uuid: "305daa9a4d874d9db0e5378e5d27b617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c292faf79e49edabdbc7a310a30503"
I20240503 14:12:43.297356 27284 raft_consensus.cc:3037] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:43.299328 27284 raft_consensus.cc:2454] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 305daa9a4d874d9db0e5378e5d27b617 in term 1.
I20240503 14:12:43.299757 27087 leader_election.cc:302] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:43.300068 28045 raft_consensus.cc:2781] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:43.300498 28045 raft_consensus.cc:686] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 LEADER]: Becoming Leader. State: Replica: 305daa9a4d874d9db0e5378e5d27b617, State: Running, Role: LEADER
I20240503 14:12:43.300776 28045 consensus_queue.cc:237] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:43.303222 26873 catalog_manager.cc:5484] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: term changed from 0 to 1, leader changed from <none> to 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "305daa9a4d874d9db0e5378e5d27b617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:43.479805 28062 raft_consensus.cc:482] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:43.480003 28062 raft_consensus.cc:504] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:43.480626 28062 leader_election.cc:288] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:43.481221 27154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cca566ca06643ffbc72f02be6d544b4" candidate_uuid: "f6c292faf79e49edabdbc7a310a30503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305daa9a4d874d9db0e5378e5d27b617" is_pre_election: true
I20240503 14:12:43.481462 27154 raft_consensus.cc:2454] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6c292faf79e49edabdbc7a310a30503 in term 0.
I20240503 14:12:43.481921 27219 leader_election.cc:302] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:43.482180 28062 raft_consensus.cc:2781] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:43.482291 28062 raft_consensus.cc:482] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:43.482359 28062 raft_consensus.cc:3037] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:43.484280 28062 raft_consensus.cc:504] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:43.484706 28062 leader_election.cc:288] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 election: Requested vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:43.485318 27154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cca566ca06643ffbc72f02be6d544b4" candidate_uuid: "f6c292faf79e49edabdbc7a310a30503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305daa9a4d874d9db0e5378e5d27b617"
I20240503 14:12:43.485499 27154 raft_consensus.cc:3037] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:43.487248 27154 raft_consensus.cc:2454] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6c292faf79e49edabdbc7a310a30503 in term 1.
I20240503 14:12:43.487661 27219 leader_election.cc:302] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:43.487993 28062 raft_consensus.cc:2781] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:43.488144 28062 raft_consensus.cc:686] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Becoming Leader. State: Replica: f6c292faf79e49edabdbc7a310a30503, State: Running, Role: LEADER
I20240503 14:12:43.488435 28062 consensus_queue.cc:237] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:43.491089 26873 catalog_manager.cc:5484] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6c292faf79e49edabdbc7a310a30503 (127.22.242.67). New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:43.646014 28045 raft_consensus.cc:482] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:43.646313 28045 raft_consensus.cc:504] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:43.647059 28045 leader_election.cc:288] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
I20240503 14:12:43.652467 27022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c47ed0ca358f4370b61529f8ffba2503" candidate_uuid: "305daa9a4d874d9db0e5378e5d27b617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9" is_pre_election: true
I20240503 14:12:43.652707 27022 raft_consensus.cc:2454] T c47ed0ca358f4370b61529f8ffba2503 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 305daa9a4d874d9db0e5378e5d27b617 in term 0.
I20240503 14:12:43.653152 27087 leader_election.cc:302] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, 820e18ddcc1e4d53821d43bc32652bc9; no voters:
I20240503 14:12:43.653492 28045 raft_consensus.cc:2781] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:43.653609 28045 raft_consensus.cc:482] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:43.653730 28045 raft_consensus.cc:3037] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:43.655593 28045 raft_consensus.cc:504] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:43.656123 28045 leader_election.cc:288] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 1 election: Requested vote from peers 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
I20240503 14:12:43.656541 27022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c47ed0ca358f4370b61529f8ffba2503" candidate_uuid: "305daa9a4d874d9db0e5378e5d27b617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9"
I20240503 14:12:43.656767 27022 raft_consensus.cc:3037] T c47ed0ca358f4370b61529f8ffba2503 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:43.658589 27022 raft_consensus.cc:2454] T c47ed0ca358f4370b61529f8ffba2503 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 305daa9a4d874d9db0e5378e5d27b617 in term 1.
I20240503 14:12:43.658937 27087 leader_election.cc:302] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, 820e18ddcc1e4d53821d43bc32652bc9; no voters:
I20240503 14:12:43.659191 28045 raft_consensus.cc:2781] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:43.659348 28045 raft_consensus.cc:686] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 LEADER]: Becoming Leader. State: Replica: 305daa9a4d874d9db0e5378e5d27b617, State: Running, Role: LEADER
I20240503 14:12:43.659622 28045 consensus_queue.cc:237] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } }
I20240503 14:12:43.662145 26873 catalog_manager.cc:5484] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: term changed from 0 to 1, leader changed from <none> to 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "305daa9a4d874d9db0e5378e5d27b617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:43.848474 28045 consensus_queue.cc:1035] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:44.066092 28135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:44.066402 28135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:44.073726 28135 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 14:12:44.107617 28062 consensus_queue.cc:1035] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:44.209273 27022 raft_consensus.cc:1261] T c47ed0ca358f4370b61529f8ffba2503 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:44.210261 28045 consensus_queue.cc:1035] T c47ed0ca358f4370b61529f8ffba2503 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 183.223 ms
time per row: 0.0916117 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:12:45.340626 26882 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56310:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:45.349468 27264 tablet_service.cc:1467] Processing CreateTablet for tablet 41b3bc788e164ce99c05a492ee768724 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6cdd58518c114d93a32a80b11512af8f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:45.349872 27002 tablet_service.cc:1467] Processing CreateTablet for tablet 0c41cab275df431f8ca3df66f9dcb871 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6cdd58518c114d93a32a80b11512af8f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:45.349938 27264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b3bc788e164ce99c05a492ee768724. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:45.349820 27001 tablet_service.cc:1467] Processing CreateTablet for tablet 41b3bc788e164ce99c05a492ee768724 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6cdd58518c114d93a32a80b11512af8f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:45.350248 27002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c41cab275df431f8ca3df66f9dcb871. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:45.350454 27001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b3bc788e164ce99c05a492ee768724. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:45.350400 27132 tablet_service.cc:1467] Processing CreateTablet for tablet 619d33242fee4c73b1abc8eafb1c00e8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6cdd58518c114d93a32a80b11512af8f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:45.350775 27132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619d33242fee4c73b1abc8eafb1c00e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:45.355242 27003 tablet_service.cc:1467] Processing CreateTablet for tablet 619d33242fee4c73b1abc8eafb1c00e8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6cdd58518c114d93a32a80b11512af8f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:45.355679 27003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619d33242fee4c73b1abc8eafb1c00e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:45.359817 28194 tablet_bootstrap.cc:492] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9: Bootstrap starting.
I20240503 14:12:45.361049 27263 tablet_service.cc:1467] Processing CreateTablet for tablet 0c41cab275df431f8ca3df66f9dcb871 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6cdd58518c114d93a32a80b11512af8f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:45.361531 28194 tablet_bootstrap.cc:654] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:45.361603 27263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c41cab275df431f8ca3df66f9dcb871. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:45.370746 28196 tablet_bootstrap.cc:492] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503: Bootstrap starting.
I20240503 14:12:45.373708 28196 tablet_bootstrap.cc:654] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:45.373816 28193 tablet_bootstrap.cc:492] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617: Bootstrap starting.
I20240503 14:12:45.375633 28196 tablet_bootstrap.cc:492] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503: No bootstrap required, opened a new log
I20240503 14:12:45.375634 28193 tablet_bootstrap.cc:654] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:45.375666 28194 tablet_bootstrap.cc:492] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9: No bootstrap required, opened a new log
I20240503 14:12:45.375777 28196 ts_tablet_manager.cc:1367] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:12:45.375798 28194 ts_tablet_manager.cc:1367] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20240503 14:12:45.376451 28196 raft_consensus.cc:348] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:45.376734 28196 raft_consensus.cc:374] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:45.376885 28196 raft_consensus.cc:729] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c292faf79e49edabdbc7a310a30503, State: Initialized, Role: FOLLOWER
I20240503 14:12:45.376719 28194 raft_consensus.cc:348] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:45.376989 28194 raft_consensus.cc:374] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:45.377198 28194 raft_consensus.cc:729] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Initialized, Role: FOLLOWER
I20240503 14:12:45.377205 28196 consensus_queue.cc:260] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:45.377532 28194 consensus_queue.cc:260] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:45.377924 28196 ts_tablet_manager.cc:1398] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:45.378223 28196 tablet_bootstrap.cc:492] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503: Bootstrap starting.
I20240503 14:12:45.379817 28196 tablet_bootstrap.cc:654] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:45.383008 28194 ts_tablet_manager.cc:1398] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:12:45.383428 28194 tablet_bootstrap.cc:492] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9: Bootstrap starting.
I20240503 14:12:45.384246 28193 tablet_bootstrap.cc:492] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617: No bootstrap required, opened a new log
I20240503 14:12:45.384373 28196 tablet_bootstrap.cc:492] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503: No bootstrap required, opened a new log
I20240503 14:12:45.384462 28193 ts_tablet_manager.cc:1367] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240503 14:12:45.384586 28196 ts_tablet_manager.cc:1367] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20240503 14:12:45.385354 28196 raft_consensus.cc:348] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:45.385362 28193 raft_consensus.cc:348] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:45.385553 28193 raft_consensus.cc:374] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:45.385643 28196 raft_consensus.cc:374] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:45.385731 28193 raft_consensus.cc:729] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305daa9a4d874d9db0e5378e5d27b617, State: Initialized, Role: FOLLOWER
I20240503 14:12:45.385738 28196 raft_consensus.cc:729] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c292faf79e49edabdbc7a310a30503, State: Initialized, Role: FOLLOWER
I20240503 14:12:45.385988 28196 consensus_queue.cc:260] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:45.385973 28193 consensus_queue.cc:260] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:45.386562 28194 tablet_bootstrap.cc:654] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:45.386585 28196 ts_tablet_manager.cc:1398] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:45.387167 28193 ts_tablet_manager.cc:1398] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:45.390046 28194 tablet_bootstrap.cc:492] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9: No bootstrap required, opened a new log
I20240503 14:12:45.390280 28194 ts_tablet_manager.cc:1367] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 14:12:45.391093 28194 raft_consensus.cc:348] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:45.391280 28194 raft_consensus.cc:374] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:45.391362 28194 raft_consensus.cc:729] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Initialized, Role: FOLLOWER
I20240503 14:12:45.391579 28194 consensus_queue.cc:260] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:45.391978 28194 ts_tablet_manager.cc:1398] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:45.392341 28194 tablet_bootstrap.cc:492] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9: Bootstrap starting.
I20240503 14:12:45.393893 28194 tablet_bootstrap.cc:654] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:45.395817 28194 tablet_bootstrap.cc:492] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9: No bootstrap required, opened a new log
I20240503 14:12:45.395983 28194 ts_tablet_manager.cc:1367] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:12:45.396673 28194 raft_consensus.cc:348] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:45.396823 28194 raft_consensus.cc:374] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:45.396890 28194 raft_consensus.cc:729] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Initialized, Role: FOLLOWER
I20240503 14:12:45.397104 28194 consensus_queue.cc:260] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:45.397516 28194 ts_tablet_manager.cc:1398] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:12:45.420534 28065 raft_consensus.cc:482] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:45.420771 28065 raft_consensus.cc:504] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:45.421358 28065 leader_election.cc:288] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:45.422048 27154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619d33242fee4c73b1abc8eafb1c00e8" candidate_uuid: "820e18ddcc1e4d53821d43bc32652bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305daa9a4d874d9db0e5378e5d27b617" is_pre_election: true
I20240503 14:12:45.422274 27154 raft_consensus.cc:2454] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 820e18ddcc1e4d53821d43bc32652bc9 in term 0.
I20240503 14:12:45.422856 26958 leader_election.cc:302] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, 820e18ddcc1e4d53821d43bc32652bc9; no voters:
I20240503 14:12:45.423151 28065 raft_consensus.cc:2781] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:45.423270 28065 raft_consensus.cc:482] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:45.423385 28065 raft_consensus.cc:3037] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:45.425434 28065 raft_consensus.cc:504] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:45.425969 28065 leader_election.cc:288] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 election: Requested vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:45.426534 27154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619d33242fee4c73b1abc8eafb1c00e8" candidate_uuid: "820e18ddcc1e4d53821d43bc32652bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305daa9a4d874d9db0e5378e5d27b617"
I20240503 14:12:45.426752 27154 raft_consensus.cc:3037] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:45.428738 27154 raft_consensus.cc:2454] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 820e18ddcc1e4d53821d43bc32652bc9 in term 1.
I20240503 14:12:45.429205 26958 leader_election.cc:302] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, 820e18ddcc1e4d53821d43bc32652bc9; no voters:
I20240503 14:12:45.429486 28065 raft_consensus.cc:2781] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:45.429909 28065 raft_consensus.cc:686] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Running, Role: LEADER
I20240503 14:12:45.430191 28065 consensus_queue.cc:237] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:45.433009 26881 catalog_manager.cc:5484] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:45.442596 28155 raft_consensus.cc:482] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:45.442770 28155 raft_consensus.cc:504] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:45.443275 28155 leader_election.cc:288] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
I20240503 14:12:45.443853 27022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b3bc788e164ce99c05a492ee768724" candidate_uuid: "f6c292faf79e49edabdbc7a310a30503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9" is_pre_election: true
I20240503 14:12:45.444052 27022 raft_consensus.cc:2454] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6c292faf79e49edabdbc7a310a30503 in term 0.
I20240503 14:12:45.444525 27217 leader_election.cc:302] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 820e18ddcc1e4d53821d43bc32652bc9, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:45.444825 28155 raft_consensus.cc:2781] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:45.444926 28155 raft_consensus.cc:482] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:45.445010 28155 raft_consensus.cc:3037] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:45.446913 28155 raft_consensus.cc:504] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:45.447340 28155 leader_election.cc:288] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 election: Requested vote from peers 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
I20240503 14:12:45.447891 27022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b3bc788e164ce99c05a492ee768724" candidate_uuid: "f6c292faf79e49edabdbc7a310a30503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9"
I20240503 14:12:45.448076 27022 raft_consensus.cc:3037] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:45.449894 27022 raft_consensus.cc:2454] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6c292faf79e49edabdbc7a310a30503 in term 1.
I20240503 14:12:45.450265 27217 leader_election.cc:302] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 820e18ddcc1e4d53821d43bc32652bc9, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:45.450584 28155 raft_consensus.cc:2781] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:45.450947 28155 raft_consensus.cc:686] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Becoming Leader. State: Replica: f6c292faf79e49edabdbc7a310a30503, State: Running, Role: LEADER
I20240503 14:12:45.451241 28155 consensus_queue.cc:237] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } }
I20240503 14:12:45.453787 26881 catalog_manager.cc:5484] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6c292faf79e49edabdbc7a310a30503 (127.22.242.67). New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:45.522151 28155 raft_consensus.cc:482] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:45.522377 28155 raft_consensus.cc:504] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:45.522887 28155 leader_election.cc:288] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
I20240503 14:12:45.523564 27022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c41cab275df431f8ca3df66f9dcb871" candidate_uuid: "f6c292faf79e49edabdbc7a310a30503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9" is_pre_election: true
I20240503 14:12:45.523784 27022 raft_consensus.cc:2454] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6c292faf79e49edabdbc7a310a30503 in term 0.
I20240503 14:12:45.524343 27217 leader_election.cc:302] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 820e18ddcc1e4d53821d43bc32652bc9, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:45.524653 28155 raft_consensus.cc:2781] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:45.524750 28155 raft_consensus.cc:482] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:45.524827 28155 raft_consensus.cc:3037] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:45.526749 28155 raft_consensus.cc:504] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:45.527268 28155 leader_election.cc:288] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 election: Requested vote from peers 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
I20240503 14:12:45.527827 27022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c41cab275df431f8ca3df66f9dcb871" candidate_uuid: "f6c292faf79e49edabdbc7a310a30503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9"
I20240503 14:12:45.528013 27022 raft_consensus.cc:3037] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:45.529930 27022 raft_consensus.cc:2454] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6c292faf79e49edabdbc7a310a30503 in term 1.
I20240503 14:12:45.530404 27217 leader_election.cc:302] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 820e18ddcc1e4d53821d43bc32652bc9, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:12:45.530685 28155 raft_consensus.cc:2781] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:45.530839 28155 raft_consensus.cc:686] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Becoming Leader. State: Replica: f6c292faf79e49edabdbc7a310a30503, State: Running, Role: LEADER
I20240503 14:12:45.531095 28155 consensus_queue.cc:237] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } }
I20240503 14:12:45.533880 26881 catalog_manager.cc:5484] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6c292faf79e49edabdbc7a310a30503 (127.22.242.67). New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:45.868921 28155 consensus_queue.cc:1035] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:45.893270 28207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:45.893600 28207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:45.901491 28207 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 14:12:45.977653 28065 consensus_queue.cc:1035] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:46.067811 28204 consensus_queue.cc:1035] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 222.45 ms
time per row: 0.111225 ms
I20240503 14:12:46.266356 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.68:41205
--local_ip_for_outbound_sockets=127.22.242.68
--tserver_master_addrs=127.22.242.126:45965
--webserver_port=39461
--webserver_interface=127.22.242.68
--builtin_ntp_servers=127.22.242.84:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:12:46.607261 28259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:46.607573 28259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:46.607721 28259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:46.616447 28259 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:46.616595 28259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:46.616771 28259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.68
I20240503 14:12:46.626526 28259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.68:41205
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.242.68
--webserver_port=39461
--tserver_master_addrs=127.22.242.126:45965
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.68
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:46.627887 28259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:46.629781 28259 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:46.641778 28265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:46.641778 28264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:46.641846 28267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:46.643013 28259 server_base.cc:1034] running on GCE node
I20240503 14:12:46.643677 28259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:46.644737 28259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:46.645968 28259 hybrid_clock.cc:648] HybridClock initialized: now 1714745566645906 us; error 53 us; skew 500 ppm
I20240503 14:12:46.646371 28259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:46.648967 28259 webserver.cc:460] Webserver started at http://127.22.242.68:39461/ using document root <none> and password file <none>
I20240503 14:12:46.649716 28259 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:46.649861 28259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:46.655365 28259 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:12:46.658334 28273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:46.659682 28259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:46.659875 28259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4165ee79ee03466981e067d998339611"
format_stamp: "Formatted at 2024-05-03 14:12:33 on dist-test-slave-r3pd"
I20240503 14:12:46.660791 28259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:46.683306 28259 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:46.684238 28259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:46.684576 28259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:46.685693 28259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:46.687237 28259 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:46.687314 28259 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:46.687419 28259 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:46.687466 28259 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:46.726235 28259 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.68:41205
I20240503 14:12:46.726270 28385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.68:41205 every 8 connection(s)
I20240503 14:12:46.728468 28259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:12:46.731161 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 28259
I20240503 14:12:46.731887 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.69:39081
--local_ip_for_outbound_sockets=127.22.242.69
--tserver_master_addrs=127.22.242.126:45965
--webserver_port=44355
--webserver_interface=127.22.242.69
--builtin_ntp_servers=127.22.242.84:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:46.745656 28386 heartbeater.cc:347] Connected to a master server at 127.22.242.126:45965
I20240503 14:12:46.746055 28386 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:46.747216 28386 heartbeater.cc:511] Master 127.22.242.126:45965 requested a full tablet report, sending...
I20240503 14:12:46.749397 26876 ts_manager.cc:194] Re-registered known tserver with Master: 4165ee79ee03466981e067d998339611 (127.22.242.68:41205)
I20240503 14:12:46.750744 26876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.68:37175
W20240503 14:12:47.088272 28390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:47.088580 28390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:47.088668 28390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:47.096079 28390 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:47.096266 28390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:47.096402 28390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.69
I20240503 14:12:47.106106 28390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.69:39081
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.242.69
--webserver_port=44355
--tserver_master_addrs=127.22.242.126:45965
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.69
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:47.107607 28390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:47.109885 28390 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:47.122751 28396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:47.123188 28395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:47.123243 28398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:47.123548 28390 server_base.cc:1034] running on GCE node
I20240503 14:12:47.124236 28390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:47.125244 28390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:47.126446 28390 hybrid_clock.cc:648] HybridClock initialized: now 1714745567126396 us; error 49 us; skew 500 ppm
I20240503 14:12:47.126780 28390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:47.129566 28390 webserver.cc:460] Webserver started at http://127.22.242.69:44355/ using document root <none> and password file <none>
I20240503 14:12:47.130378 28390 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:47.130517 28390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:47.135949 28390 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:47.139536 28404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:47.141412 28390 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:47.141604 28390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "cbb4bf53151a4061a81cf5bacf084516"
format_stamp: "Formatted at 2024-05-03 14:12:33 on dist-test-slave-r3pd"
I20240503 14:12:47.142596 28390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:47.157706 28390 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:47.158531 28390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:47.158818 28390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:47.159735 28390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:47.161314 28390 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:47.161412 28390 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:47.161545 28390 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:47.161614 28390 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:47.194007 28390 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.69:39081
I20240503 14:12:47.194023 28516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.69:39081 every 8 connection(s)
I20240503 14:12:47.195550 28390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:12:47.199990 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 28390
I20240503 14:12:47.200773 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.70:44045
--local_ip_for_outbound_sockets=127.22.242.70
--tserver_master_addrs=127.22.242.126:45965
--webserver_port=41231
--webserver_interface=127.22.242.70
--builtin_ntp_servers=127.22.242.84:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:47.210119 28517 heartbeater.cc:347] Connected to a master server at 127.22.242.126:45965
I20240503 14:12:47.210435 28517 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:47.211308 28517 heartbeater.cc:511] Master 127.22.242.126:45965 requested a full tablet report, sending...
I20240503 14:12:47.212934 26876 ts_manager.cc:194] Re-registered known tserver with Master: cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69:39081)
I20240503 14:12:47.213937 26876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.69:48269
W20240503 14:12:47.548656 28520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:47.548986 28520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:47.549134 28520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:47.557468 28520 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:47.557627 28520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:47.557825 28520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.70
I20240503 14:12:47.568629 28520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.70:44045
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.242.70
--webserver_port=41231
--tserver_master_addrs=127.22.242.126:45965
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.70
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:47.570330 28520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:47.572120 28520 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:47.585304 28526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:47.585331 28528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:47.585924 28525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:47.586194 28520 server_base.cc:1034] running on GCE node
I20240503 14:12:47.586938 28520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:47.587957 28520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:47.589169 28520 hybrid_clock.cc:648] HybridClock initialized: now 1714745567589103 us; error 71 us; skew 500 ppm
I20240503 14:12:47.589496 28520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:47.591967 28520 webserver.cc:460] Webserver started at http://127.22.242.70:41231/ using document root <none> and password file <none>
I20240503 14:12:47.592886 28520 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:47.593042 28520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:47.599383 28520 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20240503 14:12:47.602651 28534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:47.604346 28520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:47.604521 28520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "89a56aa776934c4582bfbdb0622b0cb3"
format_stamp: "Formatted at 2024-05-03 14:12:34 on dist-test-slave-r3pd"
I20240503 14:12:47.605464 28520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:47.634135 28520 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:47.635082 28520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:47.635627 28520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:47.636986 28520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:47.639349 28541 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240503 14:12:47.648432 28520 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 14:12:47.648592 28520 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.000s
I20240503 14:12:47.648777 28520 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 14:12:47.653928 28520 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 14:12:47.654076 28520 ts_tablet_manager.cc:569] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20240503 14:12:47.654471 28541 tablet_bootstrap.cc:492] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3: Bootstrap starting.
I20240503 14:12:47.674311 28541 log.cc:824] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:47.679634 28541 tablet_bootstrap.cc:492] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:47.680635 28541 tablet_bootstrap.cc:492] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3: Bootstrap complete.
I20240503 14:12:47.682366 28541 ts_tablet_manager.cc:1367] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.012s
I20240503 14:12:47.698550 28541 raft_consensus.cc:348] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } }
I20240503 14:12:47.700870 28541 raft_consensus.cc:729] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89a56aa776934c4582bfbdb0622b0cb3, State: Initialized, Role: FOLLOWER
I20240503 14:12:47.702078 28541 consensus_queue.cc:260] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } }
I20240503 14:12:47.704849 28541 ts_tablet_manager.cc:1398] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3: Time spent starting tablet: real 0.022s user 0.014s sys 0.009s
I20240503 14:12:47.729964 28520 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.70:44045
I20240503 14:12:47.729981 28649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.70:44045 every 8 connection(s)
I20240503 14:12:47.731693 28520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
W20240503 14:12:47.733275 28651 tablet.cc:2369] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:47.739921 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 28520
I20240503 14:12:47.740571 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.71:42765
--local_ip_for_outbound_sockets=127.22.242.71
--tserver_master_addrs=127.22.242.126:45965
--webserver_port=33157
--webserver_interface=127.22.242.71
--builtin_ntp_servers=127.22.242.84:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:47.743873 28650 heartbeater.cc:347] Connected to a master server at 127.22.242.126:45965
I20240503 14:12:47.744221 28650 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:47.745074 28650 heartbeater.cc:511] Master 127.22.242.126:45965 requested a full tablet report, sending...
I20240503 14:12:47.747393 26876 ts_manager.cc:194] Re-registered known tserver with Master: 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70:44045)
I20240503 14:12:47.749114 26876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.70:38207
I20240503 14:12:47.754293 28386 heartbeater.cc:503] Master 127.22.242.126:45965 was elected leader, sending a full tablet report...
W20240503 14:12:48.086072 28654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:48.086388 28654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:48.086510 28654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:48.094132 28654 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:48.094267 28654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:48.094435 28654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.71
I20240503 14:12:48.103672 28654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.71:42765
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.242.71
--webserver_port=33157
--tserver_master_addrs=127.22.242.126:45965
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.71
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:48.105156 28654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:48.106940 28654 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:48.118863 28660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:48.119005 28662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:48.118844 28659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:48.120477 28654 server_base.cc:1034] running on GCE node
I20240503 14:12:48.121255 28654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:48.122314 28654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:48.123565 28654 hybrid_clock.cc:648] HybridClock initialized: now 1714745568123504 us; error 45 us; skew 500 ppm
I20240503 14:12:48.124014 28654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:48.133738 28654 webserver.cc:460] Webserver started at http://127.22.242.71:33157/ using document root <none> and password file <none>
I20240503 14:12:48.134605 28654 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:48.134771 28654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:48.140362 28654 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:48.143647 28668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:48.145378 28654 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:48.145553 28654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0"
format_stamp: "Formatted at 2024-05-03 14:12:34 on dist-test-slave-r3pd"
I20240503 14:12:48.146414 28654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:48.163239 28654 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:48.164039 28654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:48.164367 28654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:48.165444 28654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:48.167239 28654 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:48.167335 28654 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:48.167465 28654 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:48.167515 28654 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:48.196777 28654 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.71:42765
I20240503 14:12:48.196791 28780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.71:42765 every 8 connection(s)
I20240503 14:12:48.198616 28654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:12:48.206379 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 28654
I20240503 14:12:48.206724 23497 test_util.cc:268] Using random seed: 1350115168
I20240503 14:12:48.212276 28781 heartbeater.cc:347] Connected to a master server at 127.22.242.126:45965
I20240503 14:12:48.212597 28781 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:48.213661 28781 heartbeater.cc:511] Master 127.22.242.126:45965 requested a full tablet report, sending...
I20240503 14:12:48.215494 26876 ts_manager.cc:194] Re-registered known tserver with Master: 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71:42765)
I20240503 14:12:48.216622 26876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.71:34653
I20240503 14:12:48.217046 28517 heartbeater.cc:503] Master 127.22.242.126:45965 was elected leader, sending a full tablet report...
I20240503 14:12:48.226006 23497 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 14:12:48.226539 23497 test_util.cc:268] Using random seed: 1350134980
I20240503 14:12:48.243054 23497 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 14:12:48.243495 23497 test_util.cc:268] Using random seed: 1350151910
I20240503 14:12:48.278674 23497 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240503 14:12:48.279104 23497 test_util.cc:268] Using random seed: 1350187523
I20240503 14:12:48.327730 23497 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 14:12:48.347429 23497 test_util.cc:268] Using random seed: 1350255834
I20240503 14:12:48.403439 23497 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240503 14:12:48.751694 28650 heartbeater.cc:503] Master 127.22.242.126:45965 was elected leader, sending a full tablet report...
I20240503 14:12:48.860852 28884 raft_consensus.cc:482] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:48.861227 28884 raft_consensus.cc:504] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } }
I20240503 14:12:49.005244 28884 leader_election.cc:288] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:49.049182 27151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ca4d63eab743b3977a47bcf120b3cf" candidate_uuid: "89a56aa776934c4582bfbdb0622b0cb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "305daa9a4d874d9db0e5378e5d27b617" is_pre_election: true
I20240503 14:12:49.049578 27151 raft_consensus.cc:2454] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89a56aa776934c4582bfbdb0622b0cb3 in term 1.
I20240503 14:12:49.050243 28537 leader_election.cc:302] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, 89a56aa776934c4582bfbdb0622b0cb3; no voters:
I20240503 14:12:49.051124 28884 raft_consensus.cc:2781] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:12:49.051326 28884 raft_consensus.cc:482] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:49.051468 28884 raft_consensus.cc:3037] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:49.058039 28884 raft_consensus.cc:504] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } }
I20240503 14:12:49.058923 28884 leader_election.cc:288] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [CANDIDATE]: Term 2 election: Requested vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:49.059664 27153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ca4d63eab743b3977a47bcf120b3cf" candidate_uuid: "89a56aa776934c4582bfbdb0622b0cb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "305daa9a4d874d9db0e5378e5d27b617"
I20240503 14:12:49.059875 27153 raft_consensus.cc:3037] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:49.062726 27153 raft_consensus.cc:2454] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89a56aa776934c4582bfbdb0622b0cb3 in term 2.
I20240503 14:12:49.063540 28537 leader_election.cc:302] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, 89a56aa776934c4582bfbdb0622b0cb3; no voters:
I20240503 14:12:49.066421 28884 raft_consensus.cc:2781] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:49.096680 28884 raft_consensus.cc:686] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 2 LEADER]: Becoming Leader. State: Replica: 89a56aa776934c4582bfbdb0622b0cb3, State: Running, Role: LEADER
I20240503 14:12:49.097448 28884 consensus_queue.cc:237] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } }
I20240503 14:12:49.123359 26876 catalog_manager.cc:5484] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "89a56aa776934c4582bfbdb0622b0cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:49.219717 28781 heartbeater.cc:503] Master 127.22.242.126:45965 was elected leader, sending a full tablet report...
I20240503 14:12:49.527071 27151 raft_consensus.cc:1261] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Refusing update from remote peer 89a56aa776934c4582bfbdb0622b0cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:12:49.527988 28884 consensus_queue.cc:1035] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:49.893487 27132 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:50.017072 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:50.046010 28716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:50.053109 28321 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:50.055119 28452 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:50.141513 27003 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:50.167941 27264 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:12:50.796022 27219 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.66:38751, user_credentials={real_user=slave}} blocked reactor thread for 61098.3us
I20240503 14:12:50.877944 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:50.895037 28716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:50.896126 27132 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:50.915093 28321 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:50.924804 28452 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:50.936761 27003 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:50.949782 27264 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:51.363319 27283 consensus_queue.cc:237] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 1.50, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } }
I20240503 14:12:51.367534 27020 raft_consensus.cc:1261] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240503 14:12:51.368319 28981 consensus_queue.cc:1035] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 49, Time since last communication: 0.000s
I20240503 14:12:51.372206 28981 raft_consensus.cc:2932] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } } }
I20240503 14:12:51.373195 27020 raft_consensus.cc:2932] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } } }
I20240503 14:12:51.384404 26874 catalog_manager.cc:5484] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9 reported cstate change: config changed from index -1 to 51, NON_VOTER cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } } }
W20240503 14:12:51.491531 27219 consensus_peers.cc:485] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 -> Peer cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69:39081): Couldn't send request to peer cbb4bf53151a4061a81cf5bacf084516. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c41cab275df431f8ca3df66f9dcb871. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:51.531538 27283 consensus_queue.cc:237] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 1.54, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } }
I20240503 14:12:51.539538 27023 raft_consensus.cc:1261] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240503 14:12:51.547135 28833 consensus_queue.cc:1035] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20240503 14:12:51.553258 28833 raft_consensus.cc:2932] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.55: config changed from index -1 to 55, NON_VOTER 4165ee79ee03466981e067d998339611 (127.22.242.68) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } } }
I20240503 14:12:51.558650 27023 raft_consensus.cc:2932] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index -1 to 55, NON_VOTER 4165ee79ee03466981e067d998339611 (127.22.242.68) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } } }
I20240503 14:12:51.559252 26877 catalog_manager.cc:5484] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 reported cstate change: config changed from index -1 to 55, NON_VOTER 4165ee79ee03466981e067d998339611 (127.22.242.68) added. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:51.583187 27217 consensus_peers.cc:485] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 -> Peer 4165ee79ee03466981e067d998339611 (127.22.242.68:41205): Couldn't send request to peer 4165ee79ee03466981e067d998339611. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41b3bc788e164ce99c05a492ee768724. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:51.646513 27281 consensus_queue.cc:237] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } }
I20240503 14:12:51.650502 27152 raft_consensus.cc:1261] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240503 14:12:51.651276 28833 consensus_queue.cc:1035] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240503 14:12:51.657514 28981 raft_consensus.cc:2932] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } } }
I20240503 14:12:51.657996 27154 raft_consensus.cc:2932] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } } }
I20240503 14:12:51.664445 26874 catalog_manager.cc:5484] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: config changed from index -1 to 79, NON_VOTER 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) added. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } } }
W20240503 14:12:51.686749 27220 consensus_peers.cc:485] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 -> Peer 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71:42765): Couldn't send request to peer 6b69a8c25e2b4de0bb10db8d7bdf83f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cca566ca06643ffbc72f02be6d544b4. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:51.732815 27022 consensus_queue.cc:237] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 1.60, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } }
I20240503 14:12:51.742662 27283 raft_consensus.cc:1261] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 1 index: 61. (index mismatch)
I20240503 14:12:51.743422 28882 consensus_queue.cc:1035] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20240503 14:12:51.751137 28808 raft_consensus.cc:2932] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Committing config change with OpId 1.61: config changed from index -1 to 61, NON_VOTER 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } } }
I20240503 14:12:51.752936 27283 raft_consensus.cc:2932] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Committing config change with OpId 1.61: config changed from index -1 to 61, NON_VOTER 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } } }
I20240503 14:12:51.759150 26874 catalog_manager.cc:5484] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 reported cstate change: config changed from index -1 to 61, NON_VOTER 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:51.798648 26958 consensus_peers.cc:485] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 -> Peer 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70:44045): Couldn't send request to peer 89a56aa776934c4582bfbdb0622b0cb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6385b8e7ad5849c9a17136f0967c516f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:51.827886 29037 ts_tablet_manager.cc:902] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516: Initiating tablet copy from peer f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423)
I20240503 14:12:51.886842 29037 tablet_copy_client.cc:308] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:45423
I20240503 14:12:51.895349 27283 consensus_queue.cc:237] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 1.61, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } }
I20240503 14:12:51.912870 27022 raft_consensus.cc:1261] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240503 14:12:51.918751 28902 consensus_queue.cc:1035] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
W20240503 14:12:51.919847 27217 consensus_peers.cc:485] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 -> Peer 4165ee79ee03466981e067d998339611 (127.22.242.68:41205): Couldn't send request to peer 4165ee79ee03466981e067d998339611. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab63a1d4c9764427a141feb6614fe18f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:51.920970 27304 tablet_copy_service.cc:140] P f6c292faf79e49edabdbc7a310a30503: Received BeginTabletCopySession request for tablet 0c41cab275df431f8ca3df66f9dcb871 from peer cbb4bf53151a4061a81cf5bacf084516 ({username='slave'} at 127.22.242.69:50559)
I20240503 14:12:51.921285 27304 tablet_copy_service.cc:161] P f6c292faf79e49edabdbc7a310a30503: Beginning new tablet copy session on tablet 0c41cab275df431f8ca3df66f9dcb871 from peer cbb4bf53151a4061a81cf5bacf084516 at {username='slave'} at 127.22.242.69:50559: session id = cbb4bf53151a4061a81cf5bacf084516-0c41cab275df431f8ca3df66f9dcb871
I20240503 14:12:51.931061 27304 tablet_copy_source_session.cc:205] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:51.952592 29037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c41cab275df431f8ca3df66f9dcb871. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:51.957597 28981 raft_consensus.cc:2932] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.62: config changed from index -1 to 62, NON_VOTER 4165ee79ee03466981e067d998339611 (127.22.242.68) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } } }
I20240503 14:12:51.968153 26874 catalog_manager.cc:5484] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 reported cstate change: config changed from index -1 to 62, NON_VOTER 4165ee79ee03466981e067d998339611 (127.22.242.68) added. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:51.982085 27023 raft_consensus.cc:2932] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index -1 to 62, NON_VOTER 4165ee79ee03466981e067d998339611 (127.22.242.68) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } } }
I20240503 14:12:51.992764 29037 tablet_copy_client.cc:783] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:51.993642 29037 tablet_copy_client.cc:647] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:52.004639 29037 tablet_copy_client.cc:523] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:52.035101 29037 tablet_bootstrap.cc:492] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516: Bootstrap starting.
I20240503 14:12:52.079918 29052 ts_tablet_manager.cc:902] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Initiating tablet copy from peer f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423)
I20240503 14:12:52.106027 29037 log.cc:824] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:52.128057 27022 consensus_queue.cc:237] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 1.94, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } }
I20240503 14:12:52.132987 27151 raft_consensus.cc:1261] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240503 14:12:52.135114 28963 consensus_queue.cc:1035] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20240503 14:12:52.137794 28963 raft_consensus.cc:2932] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Committing config change with OpId 1.95: config changed from index -1 to 95, NON_VOTER cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) added. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } } }
I20240503 14:12:52.142195 27154 raft_consensus.cc:2932] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index -1 to 95, NON_VOTER cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) added. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } } }
I20240503 14:12:52.149611 26874 catalog_manager.cc:5484] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: config changed from index -1 to 95, NON_VOTER cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } } }
I20240503 14:12:52.160472 29052 tablet_copy_client.cc:308] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:45423
W20240503 14:12:52.220765 26958 consensus_peers.cc:485] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 -> Peer cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69:39081): Couldn't send request to peer cbb4bf53151a4061a81cf5bacf084516. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6d0c7755fb8449ea077c3786dc619fc. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:52.222249 29061 ts_tablet_manager.cc:902] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611: Initiating tablet copy from peer f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423)
I20240503 14:12:52.233906 29063 ts_tablet_manager.cc:902] T 6385b8e7ad5849c9a17136f0967c516f P 89a56aa776934c4582bfbdb0622b0cb3: Initiating tablet copy from peer 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
I20240503 14:12:52.235661 27304 tablet_copy_service.cc:140] P f6c292faf79e49edabdbc7a310a30503: Received BeginTabletCopySession request for tablet 4cca566ca06643ffbc72f02be6d544b4 from peer 6b69a8c25e2b4de0bb10db8d7bdf83f0 ({username='slave'} at 127.22.242.71:58253)
I20240503 14:12:52.235872 27304 tablet_copy_service.cc:161] P f6c292faf79e49edabdbc7a310a30503: Beginning new tablet copy session on tablet 4cca566ca06643ffbc72f02be6d544b4 from peer 6b69a8c25e2b4de0bb10db8d7bdf83f0 at {username='slave'} at 127.22.242.71:58253: session id = 6b69a8c25e2b4de0bb10db8d7bdf83f0-4cca566ca06643ffbc72f02be6d544b4
I20240503 14:12:52.238781 27304 tablet_copy_source_session.cc:205] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:52.244751 29052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cca566ca06643ffbc72f02be6d544b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:52.275421 29052 tablet_copy_client.cc:783] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:52.276132 29052 tablet_copy_client.cc:647] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:52.303145 29063 tablet_copy_client.cc:308] T 6385b8e7ad5849c9a17136f0967c516f P 89a56aa776934c4582bfbdb0622b0cb3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:37969
I20240503 14:12:52.316424 29052 tablet_copy_client.cc:523] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:52.332979 27043 tablet_copy_service.cc:140] P 820e18ddcc1e4d53821d43bc32652bc9: Received BeginTabletCopySession request for tablet 6385b8e7ad5849c9a17136f0967c516f from peer 89a56aa776934c4582bfbdb0622b0cb3 ({username='slave'} at 127.22.242.70:58447)
I20240503 14:12:52.333340 27043 tablet_copy_service.cc:161] P 820e18ddcc1e4d53821d43bc32652bc9: Beginning new tablet copy session on tablet 6385b8e7ad5849c9a17136f0967c516f from peer 89a56aa776934c4582bfbdb0622b0cb3 at {username='slave'} at 127.22.242.70:58447: session id = 89a56aa776934c4582bfbdb0622b0cb3-6385b8e7ad5849c9a17136f0967c516f
I20240503 14:12:52.337059 27043 tablet_copy_source_session.cc:205] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:52.340420 29063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6385b8e7ad5849c9a17136f0967c516f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:52.347025 29061 tablet_copy_client.cc:308] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:45423
I20240503 14:12:52.351402 29052 tablet_bootstrap.cc:492] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Bootstrap starting.
I20240503 14:12:52.355688 27152 consensus_queue.cc:237] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } }
I20240503 14:12:52.366183 29063 tablet_copy_client.cc:783] T 6385b8e7ad5849c9a17136f0967c516f P 89a56aa776934c4582bfbdb0622b0cb3: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:52.366487 27304 tablet_copy_service.cc:140] P f6c292faf79e49edabdbc7a310a30503: Received BeginTabletCopySession request for tablet 41b3bc788e164ce99c05a492ee768724 from peer 4165ee79ee03466981e067d998339611 ({username='slave'} at 127.22.242.68:46511)
I20240503 14:12:52.366664 27304 tablet_copy_service.cc:161] P f6c292faf79e49edabdbc7a310a30503: Beginning new tablet copy session on tablet 41b3bc788e164ce99c05a492ee768724 from peer 4165ee79ee03466981e067d998339611 at {username='slave'} at 127.22.242.68:46511: session id = 4165ee79ee03466981e067d998339611-41b3bc788e164ce99c05a492ee768724
I20240503 14:12:52.366698 29063 tablet_copy_client.cc:647] T 6385b8e7ad5849c9a17136f0967c516f P 89a56aa776934c4582bfbdb0622b0cb3: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:52.369673 27304 tablet_copy_source_session.cc:205] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:52.370055 29037 tablet_bootstrap.cc:492] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516: Bootstrap replayed 1/1 log segments. Stats: ops{read=66 overwritten=0 applied=66 ignored=0} inserts{seen=697 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:52.370754 29037 tablet_bootstrap.cc:492] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516: Bootstrap complete.
I20240503 14:12:52.371357 29037 ts_tablet_manager.cc:1367] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516: Time spent bootstrapping tablet: real 0.337s user 0.136s sys 0.013s
I20240503 14:12:52.376027 29063 tablet_copy_client.cc:523] T 6385b8e7ad5849c9a17136f0967c516f P 89a56aa776934c4582bfbdb0622b0cb3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:52.376746 29061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b3bc788e164ce99c05a492ee768724. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:52.382522 29063 tablet_bootstrap.cc:492] T 6385b8e7ad5849c9a17136f0967c516f P 89a56aa776934c4582bfbdb0622b0cb3: Bootstrap starting.
I20240503 14:12:52.393790 29061 tablet_copy_client.cc:783] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:52.394258 29061 tablet_copy_client.cc:647] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:52.396497 27282 raft_consensus.cc:1261] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240503 14:12:52.397279 29000 consensus_queue.cc:1035] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240503 14:12:52.397630 29061 tablet_copy_client.cc:523] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:52.400920 29000 raft_consensus.cc:2932] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index -1 to 82, NON_VOTER 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } } }
I20240503 14:12:52.408344 29037 raft_consensus.cc:348] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } }
I20240503 14:12:52.408911 29037 raft_consensus.cc:729] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cbb4bf53151a4061a81cf5bacf084516, State: Initialized, Role: LEARNER
I20240503 14:12:52.408872 27282 raft_consensus.cc:2932] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index -1 to 82, NON_VOTER 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } } }
I20240503 14:12:52.409523 29037 consensus_queue.cc:260] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 66, Last appended: 1.66, Last appended by leader: 66, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } }
I20240503 14:12:52.410426 29061 tablet_bootstrap.cc:492] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611: Bootstrap starting.
I20240503 14:12:52.426115 26874 catalog_manager.cc:5484] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503 reported cstate change: config changed from index -1 to 82, NON_VOTER 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) added. New cstate: current_term: 1 leader_uuid: "305daa9a4d874d9db0e5378e5d27b617" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } } }
I20240503 14:12:52.439383 29052 log.cc:824] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Log is configured to *not* fsync() on all Append() calls
W20240503 14:12:52.456918 27090 consensus_peers.cc:485] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 -> Peer 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71:42765): Couldn't send request to peer 6b69a8c25e2b4de0bb10db8d7bdf83f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2db6b6ac8bb4fffb18e0b65dea27b5c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:52.457873 29061 log.cc:824] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:52.466656 29037 ts_tablet_manager.cc:1398] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516: Time spent starting tablet: real 0.095s user 0.023s sys 0.004s
I20240503 14:12:52.467965 27304 tablet_copy_service.cc:342] P f6c292faf79e49edabdbc7a310a30503: Request end of tablet copy session cbb4bf53151a4061a81cf5bacf084516-0c41cab275df431f8ca3df66f9dcb871 received from {username='slave'} at 127.22.242.69:50559
I20240503 14:12:52.468225 27304 tablet_copy_service.cc:433] P f6c292faf79e49edabdbc7a310a30503: ending tablet copy session cbb4bf53151a4061a81cf5bacf084516-0c41cab275df431f8ca3df66f9dcb871 on tablet 0c41cab275df431f8ca3df66f9dcb871 with peer cbb4bf53151a4061a81cf5bacf084516
I20240503 14:12:52.541385 29087 ts_tablet_manager.cc:902] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611: Initiating tablet copy from peer f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423)
I20240503 14:12:52.602761 29061 tablet_bootstrap.cc:492] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=69 ignored=0} inserts{seen=754 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:52.603621 29061 tablet_bootstrap.cc:492] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611: Bootstrap complete.
I20240503 14:12:52.604640 29061 ts_tablet_manager.cc:1367] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611: Time spent bootstrapping tablet: real 0.195s user 0.120s sys 0.018s
I20240503 14:12:52.612318 29061 raft_consensus.cc:348] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } }
I20240503 14:12:52.622004 29087 tablet_copy_client.cc:308] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:45423
I20240503 14:12:52.624353 29037 ts_tablet_manager.cc:902] T c6d0c7755fb8449ea077c3786dc619fc P cbb4bf53151a4061a81cf5bacf084516: Initiating tablet copy from peer 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
I20240503 14:12:52.622298 29061 raft_consensus.cc:729] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4165ee79ee03466981e067d998339611, State: Initialized, Role: LEARNER
I20240503 14:12:52.632997 27304 tablet_copy_service.cc:140] P f6c292faf79e49edabdbc7a310a30503: Received BeginTabletCopySession request for tablet ab63a1d4c9764427a141feb6614fe18f from peer 4165ee79ee03466981e067d998339611 ({username='slave'} at 127.22.242.68:46511)
I20240503 14:12:52.633188 27304 tablet_copy_service.cc:161] P f6c292faf79e49edabdbc7a310a30503: Beginning new tablet copy session on tablet ab63a1d4c9764427a141feb6614fe18f from peer 4165ee79ee03466981e067d998339611 at {username='slave'} at 127.22.242.68:46511: session id = 4165ee79ee03466981e067d998339611-ab63a1d4c9764427a141feb6614fe18f
I20240503 14:12:52.632995 29061 consensus_queue.cc:260] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 69, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } }
I20240503 14:12:52.636137 27304 tablet_copy_source_session.cc:205] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:52.638104 29087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab63a1d4c9764427a141feb6614fe18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:52.694541 29061 ts_tablet_manager.cc:1398] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611: Time spent starting tablet: real 0.090s user 0.038s sys 0.010s
I20240503 14:12:52.695510 27304 tablet_copy_service.cc:342] P f6c292faf79e49edabdbc7a310a30503: Request end of tablet copy session 4165ee79ee03466981e067d998339611-41b3bc788e164ce99c05a492ee768724 received from {username='slave'} at 127.22.242.68:46511
I20240503 14:12:52.695731 27304 tablet_copy_service.cc:433] P f6c292faf79e49edabdbc7a310a30503: ending tablet copy session 4165ee79ee03466981e067d998339611-41b3bc788e164ce99c05a492ee768724 on tablet 41b3bc788e164ce99c05a492ee768724 with peer 4165ee79ee03466981e067d998339611
I20240503 14:12:52.702811 29087 tablet_copy_client.cc:783] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:52.707417 27283 consensus_queue.cc:237] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 1.82, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } }
I20240503 14:12:52.713663 27154 raft_consensus.cc:1261] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240503 14:12:52.714319 28981 consensus_queue.cc:1035] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20240503 14:12:52.718680 28981 raft_consensus.cc:2932] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } } }
I20240503 14:12:52.722836 29087 tablet_copy_client.cc:647] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:52.724570 27154 raft_consensus.cc:2932] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } } }
I20240503 14:12:52.732782 26874 catalog_manager.cc:5484] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 reported cstate change: config changed from index -1 to 83, NON_VOTER 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:52.744446 27219 consensus_peers.cc:485] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 -> Peer 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70:44045): Couldn't send request to peer 89a56aa776934c4582bfbdb0622b0cb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efa780a04e854838a933be5934cd1578. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:52.753511 29037 tablet_copy_client.cc:308] T c6d0c7755fb8449ea077c3786dc619fc P cbb4bf53151a4061a81cf5bacf084516: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:37969
I20240503 14:12:52.753820 29087 tablet_copy_client.cc:523] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:52.757298 29087 tablet_bootstrap.cc:492] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611: Bootstrap starting.
I20240503 14:12:52.878870 28472 raft_consensus.cc:1203] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.50->[1.51-1.79] Dedup: 1.66->[1.67-1.79]
I20240503 14:12:52.908545 29113 ts_tablet_manager.cc:902] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Initiating tablet copy from peer 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:52.926069 27282 consensus_queue.cc:237] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } }
W20240503 14:12:52.928314 27220 consensus_peers.cc:485] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 -> Peer 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71:42765): Couldn't send request to peer 6b69a8c25e2b4de0bb10db8d7bdf83f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27288fd1dcc8421dab42d3c282f718ae. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:52.928539 27023 raft_consensus.cc:1261] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240503 14:12:52.931555 28902 consensus_queue.cc:1035] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240503 14:12:52.950104 27043 tablet_copy_service.cc:140] P 820e18ddcc1e4d53821d43bc32652bc9: Received BeginTabletCopySession request for tablet c6d0c7755fb8449ea077c3786dc619fc from peer cbb4bf53151a4061a81cf5bacf084516 ({username='slave'} at 127.22.242.69:55871)
I20240503 14:12:52.950349 27043 tablet_copy_service.cc:161] P 820e18ddcc1e4d53821d43bc32652bc9: Beginning new tablet copy session on tablet c6d0c7755fb8449ea077c3786dc619fc from peer cbb4bf53151a4061a81cf5bacf084516 at {username='slave'} at 127.22.242.69:55871: session id = cbb4bf53151a4061a81cf5bacf084516-c6d0c7755fb8449ea077c3786dc619fc
I20240503 14:12:52.953305 27043 tablet_copy_source_session.cc:205] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:52.954629 29037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d0c7755fb8449ea077c3786dc619fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:52.960891 28341 raft_consensus.cc:1203] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.54->[1.55-1.77] Dedup: 1.70->[1.71-1.77]
I20240503 14:12:52.965281 29020 raft_consensus.cc:2932] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index -1 to 66, NON_VOTER 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } } }
I20240503 14:12:52.974309 26877 catalog_manager.cc:5484] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 reported cstate change: config changed from index -1 to 66, NON_VOTER 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) added. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:52.984838 27023 raft_consensus.cc:2932] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index -1 to 66, NON_VOTER 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } } }
I20240503 14:12:52.989560 29037 tablet_copy_client.cc:783] T c6d0c7755fb8449ea077c3786dc619fc P cbb4bf53151a4061a81cf5bacf084516: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:52.990113 29037 tablet_copy_client.cc:647] T c6d0c7755fb8449ea077c3786dc619fc P cbb4bf53151a4061a81cf5bacf084516: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:52.993468 29037 tablet_copy_client.cc:523] T c6d0c7755fb8449ea077c3786dc619fc P cbb4bf53151a4061a81cf5bacf084516: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:53.002111 29037 tablet_bootstrap.cc:492] T c6d0c7755fb8449ea077c3786dc619fc P cbb4bf53151a4061a81cf5bacf084516: Bootstrap starting.
I20240503 14:12:53.048107 29113 tablet_copy_client.cc:308] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 6b69a8c25e2b4de0bb10db8d7bdf83f0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:38751
I20240503 14:12:53.075726 27154 consensus_queue.cc:237] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } }
I20240503 14:12:53.079799 29063 tablet_bootstrap.cc:492] T 6385b8e7ad5849c9a17136f0967c516f P 89a56aa776934c4582bfbdb0622b0cb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=69 overwritten=0 applied=69 ignored=0} inserts{seen=754 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:53.085259 29063 tablet_bootstrap.cc:492] T 6385b8e7ad5849c9a17136f0967c516f P 89a56aa776934c4582bfbdb0622b0cb3: Bootstrap complete.
I20240503 14:12:53.087507 29063 ts_tablet_manager.cc:1367] T 6385b8e7ad5849c9a17136f0967c516f P 89a56aa776934c4582bfbdb0622b0cb3: Time spent bootstrapping tablet: real 0.705s user 0.129s sys 0.052s
I20240503 14:12:53.089387 29063 raft_consensus.cc:348] T 6385b8e7ad5849c9a17136f0967c516f P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } }
I20240503 14:12:53.099129 27282 raft_consensus.cc:1261] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240503 14:12:53.099413 29063 raft_consensus.cc:729] T 6385b8e7ad5849c9a17136f0967c516f P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 89a56aa776934c4582bfbdb0622b0cb3, State: Initialized, Role: LEARNER
I20240503 14:12:53.099754 29063 consensus_queue.cc:260] T 6385b8e7ad5849c9a17136f0967c516f P 89a56aa776934c4582bfbdb0622b0cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 69, Last appended: 1.69, Last appended by leader: 69, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } }
I20240503 14:12:53.100287 29000 consensus_queue.cc:1035] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20240503 14:12:53.100951 29063 ts_tablet_manager.cc:1398] T 6385b8e7ad5849c9a17136f0967c516f P 89a56aa776934c4582bfbdb0622b0cb3: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:12:53.108348 27043 tablet_copy_service.cc:342] P 820e18ddcc1e4d53821d43bc32652bc9: Request end of tablet copy session 89a56aa776934c4582bfbdb0622b0cb3-6385b8e7ad5849c9a17136f0967c516f received from {username='slave'} at 127.22.242.70:58447
I20240503 14:12:53.108541 27043 tablet_copy_service.cc:433] P 820e18ddcc1e4d53821d43bc32652bc9: ending tablet copy session 89a56aa776934c4582bfbdb0622b0cb3-6385b8e7ad5849c9a17136f0967c516f on tablet 6385b8e7ad5849c9a17136f0967c516f with peer 89a56aa776934c4582bfbdb0622b0cb3
I20240503 14:12:53.109699 28854 raft_consensus.cc:2932] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index -1 to 105, NON_VOTER cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } } }
I20240503 14:12:53.113523 27282 raft_consensus.cc:2932] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, NON_VOTER cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } } }
I20240503 14:12:53.138449 26877 catalog_manager.cc:5484] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503 reported cstate change: config changed from index -1 to 105, NON_VOTER cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "305daa9a4d874d9db0e5378e5d27b617" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } } }
W20240503 14:12:53.141695 27217 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.65:37969, user_credentials={real_user=slave}} blocked reactor thread for 52408.6us
I20240503 14:12:53.163316 28902 raft_consensus.cc:1050] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503: attempting to promote NON_VOTER 4165ee79ee03466981e067d998339611 to VOTER
I20240503 14:12:53.167161 28902 consensus_queue.cc:237] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.79, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } }
I20240503 14:12:53.172123 27021 raft_consensus.cc:1261] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240503 14:12:53.173223 28980 raft_consensus.cc:1050] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503: attempting to promote NON_VOTER cbb4bf53151a4061a81cf5bacf084516 to VOTER
I20240503 14:12:53.174121 28341 raft_consensus.cc:1261] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [term 1 LEARNER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240503 14:12:53.175680 28980 consensus_queue.cc:237] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } }
I20240503 14:12:53.178617 29063 ts_tablet_manager.cc:902] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3: Initiating tablet copy from peer f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423)
I20240503 14:12:53.180780 27021 raft_consensus.cc:1261] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240503 14:12:53.181929 27174 tablet_copy_service.cc:140] P 305daa9a4d874d9db0e5378e5d27b617: Received BeginTabletCopySession request for tablet f2db6b6ac8bb4fffb18e0b65dea27b5c from peer 6b69a8c25e2b4de0bb10db8d7bdf83f0 ({username='slave'} at 127.22.242.71:38375)
I20240503 14:12:53.182217 27174 tablet_copy_service.cc:161] P 305daa9a4d874d9db0e5378e5d27b617: Beginning new tablet copy session on tablet f2db6b6ac8bb4fffb18e0b65dea27b5c from peer 6b69a8c25e2b4de0bb10db8d7bdf83f0 at {username='slave'} at 127.22.242.71:38375: session id = 6b69a8c25e2b4de0bb10db8d7bdf83f0-f2db6b6ac8bb4fffb18e0b65dea27b5c
I20240503 14:12:53.189695 28833 consensus_queue.cc:1035] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 78, Time since last communication: 0.000s
I20240503 14:12:53.191709 28833 consensus_queue.cc:1035] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20240503 14:12:53.192601 28472 raft_consensus.cc:1261] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516 [term 1 LEARNER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240503 14:12:53.193332 28833 consensus_queue.cc:1035] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240503 14:12:53.188112 27174 tablet_copy_source_session.cc:205] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:53.216748 28833 consensus_queue.cc:1035] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240503 14:12:53.220422 29113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2db6b6ac8bb4fffb18e0b65dea27b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:53.223306 28980 raft_consensus.cc:2932] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index 51 to 82, cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:53.226986 29113 tablet_copy_client.cc:783] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 6b69a8c25e2b4de0bb10db8d7bdf83f0: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:53.227259 29113 tablet_copy_client.cc:647] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 6b69a8c25e2b4de0bb10db8d7bdf83f0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:53.229240 29020 raft_consensus.cc:2932] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.80: config changed from index 55 to 80, 4165ee79ee03466981e067d998339611 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:12:53.239864 29063 tablet_copy_client.cc:308] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:45423
I20240503 14:12:53.240149 28472 raft_consensus.cc:2932] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516 [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 51 to 82, cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:53.242694 29113 tablet_copy_client.cc:523] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 6b69a8c25e2b4de0bb10db8d7bdf83f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:53.244091 26877 catalog_manager.cc:5484] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 reported cstate change: config changed from index 51 to 82, cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:53.247378 29113 tablet_bootstrap.cc:492] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Bootstrap starting.
I20240503 14:12:53.249569 27282 consensus_queue.cc:237] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } }
I20240503 14:12:53.256757 27021 consensus_queue.cc:237] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } }
W20240503 14:12:53.258893 26958 consensus_peers.cc:485] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 -> Peer 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70:44045): Couldn't send request to peer 89a56aa776934c4582bfbdb0622b0cb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7efea07ad3204b6e868a05e9d80f5b3a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:53.266394 26874 catalog_manager.cc:5484] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 reported cstate change: config changed from index 55 to 80, 4165ee79ee03466981e067d998339611 (127.22.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:53.267256 27021 raft_consensus.cc:2932] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 55 to 80, 4165ee79ee03466981e067d998339611 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:12:53.268535 28341 raft_consensus.cc:2932] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 55 to 80, 4165ee79ee03466981e067d998339611 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:12:53.271023 27023 raft_consensus.cc:2932] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 51 to 82, cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:53.273234 27281 raft_consensus.cc:1261] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240503 14:12:53.275004 28963 consensus_queue.cc:1035] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 70, Time since last communication: 0.000s
I20240503 14:12:53.280488 28605 raft_consensus.cc:1203] T 6385b8e7ad5849c9a17136f0967c516f P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.60->[1.61-1.74] Dedup: 1.69->[1.70-1.74]
I20240503 14:12:53.282414 28471 raft_consensus.cc:1261] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 14:12:53.286903 28963 raft_consensus.cc:2932] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index -1 to 72, NON_VOTER 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) added. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } } }
I20240503 14:12:53.288728 29052 tablet_bootstrap.cc:492] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=95 overwritten=0 applied=94 ignored=0} inserts{seen=725 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:53.291162 29052 tablet_bootstrap.cc:492] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Bootstrap complete.
I20240503 14:12:53.292781 29052 ts_tablet_manager.cc:1367] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Time spent bootstrapping tablet: real 0.942s user 0.200s sys 0.027s
I20240503 14:12:53.294083 26877 catalog_manager.cc:5484] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 reported cstate change: config changed from index -1 to 72, NON_VOTER 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:53.296375 28833 consensus_queue.cc:1035] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
W20240503 14:12:53.298218 27089 consensus_peers.cc:485] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 -> Peer cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69:39081): Couldn't send request to peer cbb4bf53151a4061a81cf5bacf084516. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f24cc60aba81429ca7275571faf77cb9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:53.300971 27281 raft_consensus.cc:2932] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index -1 to 72, NON_VOTER 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) added. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } } }
I20240503 14:12:53.303259 27304 tablet_copy_service.cc:140] P f6c292faf79e49edabdbc7a310a30503: Received BeginTabletCopySession request for tablet efa780a04e854838a933be5934cd1578 from peer 89a56aa776934c4582bfbdb0622b0cb3 ({username='slave'} at 127.22.242.70:56359)
I20240503 14:12:53.303426 27304 tablet_copy_service.cc:161] P f6c292faf79e49edabdbc7a310a30503: Beginning new tablet copy session on tablet efa780a04e854838a933be5934cd1578 from peer 89a56aa776934c4582bfbdb0622b0cb3 at {username='slave'} at 127.22.242.70:56359: session id = 89a56aa776934c4582bfbdb0622b0cb3-efa780a04e854838a933be5934cd1578
I20240503 14:12:53.306416 27304 tablet_copy_source_session.cc:205] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:53.308790 29063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa780a04e854838a933be5934cd1578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:53.316872 29063 tablet_copy_client.cc:783] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:53.317183 29063 tablet_copy_client.cc:647] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:53.321846 29020 raft_consensus.cc:2932] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index 82 to 84, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:53.345453 26866 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0c41cab275df431f8ca3df66f9dcb871 with cas_config_opid_index 82: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:53.345155 28471 raft_consensus.cc:2932] T 0c41cab275df431f8ca3df66f9dcb871 P cbb4bf53151a4061a81cf5bacf084516 [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 82 to 84, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:53.347924 29063 tablet_copy_client.cc:523] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:53.352219 29063 tablet_bootstrap.cc:492] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3: Bootstrap starting.
I20240503 14:12:53.360236 26877 catalog_manager.cc:5484] T 0c41cab275df431f8ca3df66f9dcb871 P f6c292faf79e49edabdbc7a310a30503 reported cstate change: config changed from index 82 to 84, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:53.369094 27003 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c41cab275df431f8ca3df66f9dcb871 with delete_type TABLET_DATA_TOMBSTONED (TS 820e18ddcc1e4d53821d43bc32652bc9 not found in new config with opid_index 84) from {username='slave'} at 127.0.0.1:40428
I20240503 14:12:53.381196 29159 ts_tablet_manager.cc:902] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Initiating tablet copy from peer f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423)
I20240503 14:12:53.380278 29052 raft_consensus.cc:348] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } }
I20240503 14:12:53.402210 29160 tablet_replica.cc:330] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9: stopping tablet replica
I20240503 14:12:53.403208 29160 raft_consensus.cc:2229] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:53.404521 29160 raft_consensus.cc:2258] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:53.410475 29160 ts_tablet_manager.cc:1875] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:53.428562 29087 tablet_bootstrap.cc:492] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611: Bootstrap replayed 1/1 log segments. Stats: ops{read=69 overwritten=0 applied=69 ignored=0} inserts{seen=700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:53.429438 29087 tablet_bootstrap.cc:492] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611: Bootstrap complete.
I20240503 14:12:53.429869 29087 ts_tablet_manager.cc:1367] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611: Time spent bootstrapping tablet: real 0.673s user 0.161s sys 0.018s
I20240503 14:12:53.430233 29160 ts_tablet_manager.cc:1888] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.83
I20240503 14:12:53.430809 29160 log.cc:1196] T 0c41cab275df431f8ca3df66f9dcb871 P 820e18ddcc1e4d53821d43bc32652bc9: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/0c41cab275df431f8ca3df66f9dcb871
I20240503 14:12:53.431048 29087 raft_consensus.cc:348] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } }
I20240503 14:12:53.431324 29087 raft_consensus.cc:729] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4165ee79ee03466981e067d998339611, State: Initialized, Role: LEARNER
I20240503 14:12:53.431557 29087 consensus_queue.cc:260] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 69, Last appended: 1.69, Last appended by leader: 69, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } }
I20240503 14:12:53.432117 29087 ts_tablet_manager.cc:1398] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:53.432786 26866 catalog_manager.cc:4830] TS 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969): tablet 0c41cab275df431f8ca3df66f9dcb871 (table rebalance_test_table_4 [id=6cdd58518c114d93a32a80b11512af8f]) successfully deleted
I20240503 14:12:53.437873 29052 raft_consensus.cc:729] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6b69a8c25e2b4de0bb10db8d7bdf83f0, State: Initialized, Role: LEARNER
I20240503 14:12:53.439157 29052 consensus_queue.cc:260] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 94, Last appended: 1.95, Last appended by leader: 95, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } }
I20240503 14:12:53.439543 28808 raft_consensus.cc:1050] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9: attempting to promote NON_VOTER 89a56aa776934c4582bfbdb0622b0cb3 to VOTER
I20240503 14:12:53.440461 28808 consensus_queue.cc:237] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.75, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } }
I20240503 14:12:53.443405 28605 raft_consensus.cc:1261] T 6385b8e7ad5849c9a17136f0967c516f P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 LEARNER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 14:12:53.445852 28963 consensus_queue.cc:1035] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 74, Time since last communication: 0.000s
I20240503 14:12:53.451042 29052 ts_tablet_manager.cc:1398] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Time spent starting tablet: real 0.153s user 0.014s sys 0.001s
I20240503 14:12:53.450810 28235 raft_consensus.cc:2932] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Committing config change with OpId 1.76: config changed from index 61 to 76, 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:12:53.455788 27304 tablet_copy_service.cc:342] P f6c292faf79e49edabdbc7a310a30503: Request end of tablet copy session 4165ee79ee03466981e067d998339611-ab63a1d4c9764427a141feb6614fe18f received from {username='slave'} at 127.22.242.68:46511
I20240503 14:12:53.455953 27304 tablet_copy_service.cc:433] P f6c292faf79e49edabdbc7a310a30503: ending tablet copy session 4165ee79ee03466981e067d998339611-ab63a1d4c9764427a141feb6614fe18f on tablet ab63a1d4c9764427a141feb6614fe18f with peer 4165ee79ee03466981e067d998339611
I20240503 14:12:53.461120 26877 catalog_manager.cc:5484] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 reported cstate change: config changed from index 61 to 76, 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:53.469210 28605 raft_consensus.cc:2932] T 6385b8e7ad5849c9a17136f0967c516f P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index 61 to 76, 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:12:53.473424 27023 consensus_queue.cc:237] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.76, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } }
I20240503 14:12:53.475558 27304 tablet_copy_service.cc:342] P f6c292faf79e49edabdbc7a310a30503: Request end of tablet copy session 6b69a8c25e2b4de0bb10db8d7bdf83f0-4cca566ca06643ffbc72f02be6d544b4 received from {username='slave'} at 127.22.242.71:58253
I20240503 14:12:53.475735 27304 tablet_copy_service.cc:433] P f6c292faf79e49edabdbc7a310a30503: ending tablet copy session 6b69a8c25e2b4de0bb10db8d7bdf83f0-4cca566ca06643ffbc72f02be6d544b4 on tablet 4cca566ca06643ffbc72f02be6d544b4 with peer 6b69a8c25e2b4de0bb10db8d7bdf83f0
I20240503 14:12:53.477180 28605 raft_consensus.cc:1261] T 6385b8e7ad5849c9a17136f0967c516f P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
W20240503 14:12:53.483686 27219 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.66:38751, user_credentials={real_user=slave}} blocked reactor thread for 46091.6us
I20240503 14:12:53.486635 28808 consensus_queue.cc:1035] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240503 14:12:53.506117 28808 raft_consensus.cc:2932] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Committing config change with OpId 1.77: config changed from index 76 to 77, VOTER f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) evicted. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:12:53.521700 27282 raft_consensus.cc:1261] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 14:12:53.524147 28341 raft_consensus.cc:1203] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.61->[1.62-1.77] Dedup: 1.69->[1.70-1.77]
I20240503 14:12:53.560492 29113 tablet_bootstrap.cc:492] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=97 overwritten=0 applied=97 ignored=0} inserts{seen=782 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:53.561055 29113 tablet_bootstrap.cc:492] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Bootstrap complete.
I20240503 14:12:53.561376 29113 ts_tablet_manager.cc:1367] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Time spent bootstrapping tablet: real 0.314s user 0.193s sys 0.037s
I20240503 14:12:53.562511 29113 raft_consensus.cc:348] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } }
I20240503 14:12:53.562774 29113 raft_consensus.cc:729] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6b69a8c25e2b4de0bb10db8d7bdf83f0, State: Initialized, Role: LEARNER
I20240503 14:12:53.563015 29113 consensus_queue.cc:260] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 97, Last appended: 1.97, Last appended by leader: 97, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } }
I20240503 14:12:53.564424 29113 ts_tablet_manager.cc:1398] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:53.568437 27174 tablet_copy_service.cc:342] P 305daa9a4d874d9db0e5378e5d27b617: Request end of tablet copy session 6b69a8c25e2b4de0bb10db8d7bdf83f0-f2db6b6ac8bb4fffb18e0b65dea27b5c received from {username='slave'} at 127.22.242.71:38375
I20240503 14:12:53.568686 27174 tablet_copy_service.cc:433] P 305daa9a4d874d9db0e5378e5d27b617: ending tablet copy session 6b69a8c25e2b4de0bb10db8d7bdf83f0-f2db6b6ac8bb4fffb18e0b65dea27b5c on tablet f2db6b6ac8bb4fffb18e0b65dea27b5c with peer 6b69a8c25e2b4de0bb10db8d7bdf83f0
I20240503 14:12:53.582551 29173 ts_tablet_manager.cc:902] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516: Initiating tablet copy from peer 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:53.598773 28605 consensus_queue.cc:237] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } }
I20240503 14:12:53.604211 26866 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6385b8e7ad5849c9a17136f0967c516f with cas_config_opid_index 76: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 14:12:53.603605 26956 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.67:45423, user_credentials={real_user=slave}} blocked reactor thread for 80018.4us
W20240503 14:12:53.625954 26958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.66:38751, user_credentials={real_user=slave}} blocked reactor thread for 108115us
I20240503 14:12:53.627362 26877 catalog_manager.cc:5484] T 6385b8e7ad5849c9a17136f0967c516f P 820e18ddcc1e4d53821d43bc32652bc9 reported cstate change: config changed from index 76 to 77, VOTER f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:53.631520 28604 raft_consensus.cc:2932] T 6385b8e7ad5849c9a17136f0967c516f P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index 76 to 77, VOTER f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) evicted. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:12:53.635016 29159 tablet_copy_client.cc:308] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:45423
I20240503 14:12:53.635784 27264 tablet_service.cc:1514] Processing DeleteTablet for tablet 6385b8e7ad5849c9a17136f0967c516f with delete_type TABLET_DATA_TOMBSTONED (TS f6c292faf79e49edabdbc7a310a30503 not found in new config with opid_index 77) from {username='slave'} at 127.0.0.1:41116
I20240503 14:12:53.636898 27304 tablet_copy_service.cc:140] P f6c292faf79e49edabdbc7a310a30503: Received BeginTabletCopySession request for tablet 27288fd1dcc8421dab42d3c282f718ae from peer 6b69a8c25e2b4de0bb10db8d7bdf83f0 ({username='slave'} at 127.22.242.71:58253)
I20240503 14:12:53.637081 27304 tablet_copy_service.cc:161] P f6c292faf79e49edabdbc7a310a30503: Beginning new tablet copy session on tablet 27288fd1dcc8421dab42d3c282f718ae from peer 6b69a8c25e2b4de0bb10db8d7bdf83f0 at {username='slave'} at 127.22.242.71:58253: session id = 6b69a8c25e2b4de0bb10db8d7bdf83f0-27288fd1dcc8421dab42d3c282f718ae
I20240503 14:12:53.640969 27154 raft_consensus.cc:1261] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Refusing update from remote peer 89a56aa776934c4582bfbdb0622b0cb3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:12:53.643841 28919 consensus_queue.cc:1035] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:12:53.651568 27304 tablet_copy_source_session.cc:205] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:53.686408 29183 tablet_replica.cc:330] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503: stopping tablet replica
I20240503 14:12:53.686959 29037 tablet_bootstrap.cc:492] T c6d0c7755fb8449ea077c3786dc619fc P cbb4bf53151a4061a81cf5bacf084516: Bootstrap replayed 1/1 log segments. Stats: ops{read=111 overwritten=0 applied=111 ignored=0} inserts{seen=783 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:53.687495 29037 tablet_bootstrap.cc:492] T c6d0c7755fb8449ea077c3786dc619fc P cbb4bf53151a4061a81cf5bacf084516: Bootstrap complete.
I20240503 14:12:53.687816 29037 ts_tablet_manager.cc:1367] T c6d0c7755fb8449ea077c3786dc619fc P cbb4bf53151a4061a81cf5bacf084516: Time spent bootstrapping tablet: real 0.686s user 0.170s sys 0.013s
I20240503 14:12:53.688894 29037 raft_consensus.cc:348] T c6d0c7755fb8449ea077c3786dc619fc P cbb4bf53151a4061a81cf5bacf084516 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } }
I20240503 14:12:53.689146 29037 raft_consensus.cc:729] T c6d0c7755fb8449ea077c3786dc619fc P cbb4bf53151a4061a81cf5bacf084516 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cbb4bf53151a4061a81cf5bacf084516, State: Initialized, Role: LEARNER
I20240503 14:12:53.689379 29037 consensus_queue.cc:260] T c6d0c7755fb8449ea077c3786dc619fc P cbb4bf53151a4061a81cf5bacf084516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 111, Last appended: 1.111, Last appended by leader: 111, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } }
I20240503 14:12:53.689306 28919 raft_consensus.cc:2932] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } } }
I20240503 14:12:53.690011 29037 ts_tablet_manager.cc:1398] T c6d0c7755fb8449ea077c3786dc619fc P cbb4bf53151a4061a81cf5bacf084516: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 14:12:53.691956 27043 tablet_copy_service.cc:342] P 820e18ddcc1e4d53821d43bc32652bc9: Request end of tablet copy session cbb4bf53151a4061a81cf5bacf084516-c6d0c7755fb8449ea077c3786dc619fc received from {username='slave'} at 127.22.242.69:55871
I20240503 14:12:53.692101 27043 tablet_copy_service.cc:433] P 820e18ddcc1e4d53821d43bc32652bc9: ending tablet copy session cbb4bf53151a4061a81cf5bacf084516-c6d0c7755fb8449ea077c3786dc619fc on tablet c6d0c7755fb8449ea077c3786dc619fc with peer cbb4bf53151a4061a81cf5bacf084516
I20240503 14:12:53.694696 27154 raft_consensus.cc:2932] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } } }
I20240503 14:12:53.705507 26877 catalog_manager.cc:5484] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: config changed from index -1 to 3, NON_VOTER 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) added. New cstate: current_term: 2 leader_uuid: "89a56aa776934c4582bfbdb0622b0cb3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } } }
I20240503 14:12:53.721225 29159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27288fd1dcc8421dab42d3c282f718ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:53.742256 29159 tablet_copy_client.cc:783] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:53.744457 29159 tablet_copy_client.cc:647] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:53.776208 28736 raft_consensus.cc:1203] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.78->[1.79-1.120] Dedup: 1.95->[1.96-1.120]
I20240503 14:12:53.781842 28471 raft_consensus.cc:1203] T c6d0c7755fb8449ea077c3786dc619fc P cbb4bf53151a4061a81cf5bacf084516 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.94->[1.95-1.122] Dedup: 1.111->[1.112-1.122]
I20240503 14:12:53.796458 29159 tablet_copy_client.cc:523] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:53.799902 29159 tablet_bootstrap.cc:492] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Bootstrap starting.
I20240503 14:12:53.795346 28833 raft_consensus.cc:1050] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503: attempting to promote NON_VOTER 4165ee79ee03466981e067d998339611 to VOTER
I20240503 14:12:53.795598 29143 raft_consensus.cc:1050] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503: attempting to promote NON_VOTER 4165ee79ee03466981e067d998339611 to VOTER
I20240503 14:12:53.803975 29143 consensus_queue.cc:237] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.79, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } }
W20240503 14:12:53.806118 28833 raft_consensus.cc:1054] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503: Unable to promote non-voter 4165ee79ee03466981e067d998339611: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } }.
Pending config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } }
I20240503 14:12:53.807963 29183 raft_consensus.cc:2229] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:53.811731 28340 raft_consensus.cc:1261] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611 [term 1 LEARNER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
W20240503 14:12:53.818285 28538 consensus_peers.cc:485] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 -> Peer 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71:42765): Couldn't send request to peer 6b69a8c25e2b4de0bb10db8d7bdf83f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09ca4d63eab743b3977a47bcf120b3cf. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:53.820895 27021 raft_consensus.cc:1261] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240503 14:12:53.858076 29194 ts_tablet_manager.cc:902] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3: Initiating tablet copy from peer 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
I20240503 14:12:53.823909 29147 consensus_queue.cc:1035] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 78, Time since last communication: 0.000s
I20240503 14:12:53.875012 29183 pending_rounds.cc:62] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503: Trying to abort 1 pending ops.
I20240503 14:12:53.875106 28833 consensus_queue.cc:1035] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 78, Time since last communication: 0.000s
I20240503 14:12:53.875197 29183 pending_rounds.cc:68] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503: Aborting op as it isn't in flight: id { term: 1 index: 75 } timestamp: 7023597868726153216 op_type: WRITE_OP write_request { tablet_id: "6385b8e7ad5849c9a17136f0967c516f" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\034K\244b\331\256\020,\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""c141cd298e08c06f" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023597868622381056 authz_token { token_data: "\010\214\344\323\261\006\"3\n\005slave\022*\n 8c704c1747ee4a41b347496dc856e7bf\020\001\030\001 \001(\001" signature: "<redacted>""\325.-%`\262\034\360\234\025j\204\277>k\243\357\210\023\273\304\266\023\321y\335\375\001\373\242tt\255CQ+\362x\027@f\342u:lS\177cO\356J\031\'\261\366\224\366\267\003\200W\345\2134" signing_key_seq_num: 0 } } request_id { client_id: "c348d9cb2bd74014af331153324a8011" seq_no: 226 first_incomplete_seq_no: 226 attempt_no: 0 }
I20240503 14:12:53.876022 29183 raft_consensus.cc:2258] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:53.878119 29183 ts_tablet_manager.cc:1875] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:53.880839 28341 raft_consensus.cc:2932] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 62 to 80, 4165ee79ee03466981e067d998339611 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:12:53.880053 27021 raft_consensus.cc:2932] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 62 to 80, 4165ee79ee03466981e067d998339611 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:12:53.889689 26875 catalog_manager.cc:5484] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611 reported cstate change: config changed from index 62 to 80, 4165ee79ee03466981e067d998339611 (127.22.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:12:53.894737 29173 tablet_copy_client.cc:308] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:38751
I20240503 14:12:53.916430 28735 raft_consensus.cc:1203] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.81->[1.82-1.103] Dedup: 1.97->[1.98-1.103]
I20240503 14:12:53.930871 27174 tablet_copy_service.cc:140] P 305daa9a4d874d9db0e5378e5d27b617: Received BeginTabletCopySession request for tablet f24cc60aba81429ca7275571faf77cb9 from peer cbb4bf53151a4061a81cf5bacf084516 ({username='slave'} at 127.22.242.69:60405)
I20240503 14:12:53.931069 27174 tablet_copy_service.cc:161] P 305daa9a4d874d9db0e5378e5d27b617: Beginning new tablet copy session on tablet f24cc60aba81429ca7275571faf77cb9 from peer cbb4bf53151a4061a81cf5bacf084516 at {username='slave'} at 127.22.242.69:60405: session id = cbb4bf53151a4061a81cf5bacf084516-f24cc60aba81429ca7275571faf77cb9
I20240503 14:12:53.933707 28808 raft_consensus.cc:1050] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9: attempting to promote NON_VOTER cbb4bf53151a4061a81cf5bacf084516 to VOTER
I20240503 14:12:53.933916 27174 tablet_copy_source_session.cc:205] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:53.934690 28808 consensus_queue.cc:237] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 1.123, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } }
I20240503 14:12:53.937635 29173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f24cc60aba81429ca7275571faf77cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:53.945327 28471 raft_consensus.cc:1261] T c6d0c7755fb8449ea077c3786dc619fc P cbb4bf53151a4061a81cf5bacf084516 [term 1 LEARNER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 123. Preceding OpId from leader: term: 1 index: 124. (index mismatch)
I20240503 14:12:53.878615 28833 raft_consensus.cc:2932] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.80: config changed from index 62 to 80, 4165ee79ee03466981e067d998339611 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:12:53.947954 27154 raft_consensus.cc:1261] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 123. Preceding OpId from leader: term: 1 index: 124. (index mismatch)
I20240503 14:12:53.950489 29171 consensus_queue.cc:1035] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20240503 14:12:53.953197 29194 tablet_copy_client.cc:308] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:37969
I20240503 14:12:53.954241 27043 tablet_copy_service.cc:140] P 820e18ddcc1e4d53821d43bc32652bc9: Received BeginTabletCopySession request for tablet 7efea07ad3204b6e868a05e9d80f5b3a from peer 89a56aa776934c4582bfbdb0622b0cb3 ({username='slave'} at 127.22.242.70:58447)
I20240503 14:12:53.954387 27043 tablet_copy_service.cc:161] P 820e18ddcc1e4d53821d43bc32652bc9: Beginning new tablet copy session on tablet 7efea07ad3204b6e868a05e9d80f5b3a from peer 89a56aa776934c4582bfbdb0622b0cb3 at {username='slave'} at 127.22.242.70:58447: session id = 89a56aa776934c4582bfbdb0622b0cb3-7efea07ad3204b6e868a05e9d80f5b3a
I20240503 14:12:53.957703 27043 tablet_copy_source_session.cc:205] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:53.959534 29171 consensus_queue.cc:1035] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20240503 14:12:53.960754 28882 raft_consensus.cc:2932] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Committing config change with OpId 1.124: config changed from index 95 to 124, cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:53.963742 27154 raft_consensus.cc:2932] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Committing config change with OpId 1.124: config changed from index 95 to 124, cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:53.970718 28471 raft_consensus.cc:2932] T c6d0c7755fb8449ea077c3786dc619fc P cbb4bf53151a4061a81cf5bacf084516 [term 1 FOLLOWER]: Committing config change with OpId 1.124: config changed from index 95 to 124, cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:53.976461 26877 catalog_manager.cc:5484] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: config changed from index 95 to 124, cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:53.982756 27282 consensus_queue.cc:237] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.82, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } }
I20240503 14:12:53.989528 27023 consensus_queue.cc:237] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 1.125, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } }
I20240503 14:12:53.991245 28340 raft_consensus.cc:1261] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240503 14:12:53.996301 28833 consensus_queue.cc:1035] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 81, Time since last communication: 0.000s
I20240503 14:12:53.999217 29143 raft_consensus.cc:2932] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.83: config changed from index 80 to 83, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:12:54.003484 28471 raft_consensus.cc:1261] T c6d0c7755fb8449ea077c3786dc619fc P cbb4bf53151a4061a81cf5bacf084516 [term 1 FOLLOWER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240503 14:12:54.005250 26866 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ab63a1d4c9764427a141feb6614fe18f with cas_config_opid_index 80: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:54.007401 26874 catalog_manager.cc:5484] T ab63a1d4c9764427a141feb6614fe18f P f6c292faf79e49edabdbc7a310a30503 reported cstate change: config changed from index 80 to 83, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:54.009507 28339 raft_consensus.cc:2932] T ab63a1d4c9764427a141feb6614fe18f P 4165ee79ee03466981e067d998339611 [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index 80 to 83, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:12:54.013535 29173 tablet_copy_client.cc:783] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:54.014122 28882 consensus_queue.cc:1035] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20240503 14:12:54.014118 27003 tablet_service.cc:1514] Processing DeleteTablet for tablet ab63a1d4c9764427a141feb6614fe18f with delete_type TABLET_DATA_TOMBSTONED (TS 820e18ddcc1e4d53821d43bc32652bc9 not found in new config with opid_index 83) from {username='slave'} at 127.0.0.1:40428
I20240503 14:12:54.025744 29170 raft_consensus.cc:2932] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Committing config change with OpId 1.126: config changed from index 124 to 126, VOTER 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) evicted. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:54.032366 26866 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6d0c7755fb8449ea077c3786dc619fc with cas_config_opid_index 124: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:54.013926 29173 tablet_copy_client.cc:647] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:54.037317 26875 catalog_manager.cc:5484] T c6d0c7755fb8449ea077c3786dc619fc P 820e18ddcc1e4d53821d43bc32652bc9 reported cstate change: config changed from index 124 to 126, VOTER 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:54.038450 28471 raft_consensus.cc:2932] T c6d0c7755fb8449ea077c3786dc619fc P cbb4bf53151a4061a81cf5bacf084516 [term 1 FOLLOWER]: Committing config change with OpId 1.126: config changed from index 124 to 126, VOTER 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) evicted. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:54.045684 27132 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d0c7755fb8449ea077c3786dc619fc with delete_type TABLET_DATA_TOMBSTONED (TS 305daa9a4d874d9db0e5378e5d27b617 not found in new config with opid_index 126) from {username='slave'} at 127.0.0.1:54420
I20240503 14:12:54.054374 29173 tablet_copy_client.cc:523] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:54.058029 29215 tablet_replica.cc:330] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9: stopping tablet replica
I20240503 14:12:54.062353 29215 raft_consensus.cc:2229] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:54.062660 29215 pending_rounds.cc:62] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9: Trying to abort 1 pending ops.
I20240503 14:12:54.064225 29113 ts_tablet_manager.cc:902] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Initiating tablet copy from peer 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70:44045)
I20240503 14:12:54.064517 29173 tablet_bootstrap.cc:492] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516: Bootstrap starting.
I20240503 14:12:54.062840 29215 pending_rounds.cc:68] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9: Aborting op as it isn't in flight: id { term: 1 index: 82 } timestamp: 7023597870996824064 op_type: WRITE_OP write_request { tablet_id: "ab63a1d4c9764427a141feb6614fe18f" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000k\226&9]R^\200\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""709f81daf00c91f3" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023597870631587840 authz_token { token_data: "\010\214\344\323\261\006\"3\n\005slave\022*\n 8c704c1747ee4a41b347496dc856e7bf\020\001\030\001 \001(\001" signature: "<redacted>""\325.-%`\262\034\360\234\025j\204\277>k\243\357\210\023\273\304\266\023\321y\335\375\001\373\242tt\255CQ+\362x\027@f\342u:lS\177cO\356J\031\'\261\366\224\366\267\003\200W\345\2134" signing_key_seq_num: 0 } } request_id { client_id: "c348d9cb2bd74014af331153324a8011" seq_no: 235 first_incomplete_seq_no: 235 attempt_no: 0 }
I20240503 14:12:54.068413 29215 raft_consensus.cc:2258] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:54.069710 29215 ts_tablet_manager.cc:1875] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:54.073993 29147 raft_consensus.cc:1050] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503: attempting to promote NON_VOTER 6b69a8c25e2b4de0bb10db8d7bdf83f0 to VOTER
I20240503 14:12:54.074728 29147 consensus_queue.cc:237] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 1.123, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } }
I20240503 14:12:54.077790 29194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7efea07ad3204b6e868a05e9d80f5b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:54.077939 28734 raft_consensus.cc:1261] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 LEARNER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 124. (index mismatch)
I20240503 14:12:54.078830 27154 raft_consensus.cc:1261] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 123. Preceding OpId from leader: term: 1 index: 124. (index mismatch)
I20240503 14:12:54.079938 29143 consensus_queue.cc:1035] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20240503 14:12:54.083837 29143 consensus_queue.cc:1035] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 122, Time since last communication: 0.000s
I20240503 14:12:54.086266 28902 raft_consensus.cc:2932] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.124: config changed from index 79 to 124, 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:54.086570 28735 raft_consensus.cc:1203] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.122->[1.123-1.124] Dedup: 1.123->[1.124-1.124]
I20240503 14:12:54.091955 28854 raft_consensus.cc:1050] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617: attempting to promote NON_VOTER 6b69a8c25e2b4de0bb10db8d7bdf83f0 to VOTER
I20240503 14:12:54.093112 28854 consensus_queue.cc:237] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } }
I20240503 14:12:54.096097 28735 raft_consensus.cc:1261] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 LEARNER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240503 14:12:54.096973 28999 consensus_queue.cc:1035] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 103, Time since last communication: 0.000s
I20240503 14:12:54.099474 28999 raft_consensus.cc:2932] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index 82 to 105, 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:54.100195 26875 catalog_manager.cc:5484] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 reported cstate change: config changed from index 79 to 124, 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:54.100757 29216 tablet_replica.cc:330] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617: stopping tablet replica
I20240503 14:12:54.101243 29216 raft_consensus.cc:2229] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:54.101653 29216 raft_consensus.cc:2258] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:54.105497 26874 catalog_manager.cc:5484] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: config changed from index 82 to 105, 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "305daa9a4d874d9db0e5378e5d27b617" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:54.107568 27151 raft_consensus.cc:2932] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Committing config change with OpId 1.124: config changed from index 79 to 124, 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:54.104202 28736 raft_consensus.cc:2932] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 82 to 105, 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:54.109622 29216 ts_tablet_manager.cc:1875] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:54.113158 29183 ts_tablet_manager.cc:1888] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.75
I20240503 14:12:54.117561 27281 consensus_queue.cc:237] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 1.125, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } }
I20240503 14:12:54.117815 29183 log.cc:1196] T 6385b8e7ad5849c9a17136f0967c516f P f6c292faf79e49edabdbc7a310a30503: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/6385b8e7ad5849c9a17136f0967c516f
I20240503 14:12:54.119740 26866 catalog_manager.cc:4830] TS f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423): tablet 6385b8e7ad5849c9a17136f0967c516f (table rebalance_test_table_2 [id=8c704c1747ee4a41b347496dc856e7bf]) successfully deleted
I20240503 14:12:54.119997 29194 tablet_copy_client.cc:783] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:54.120347 29194 tablet_copy_client.cc:647] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:54.123148 29194 tablet_copy_client.cc:523] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:54.123529 28735 raft_consensus.cc:1261] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 126. (index mismatch)
I20240503 14:12:54.124646 27154 consensus_queue.cc:237] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } }
I20240503 14:12:54.124482 28736 raft_consensus.cc:2932] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 FOLLOWER]: Committing config change with OpId 1.124: config changed from index 79 to 124, 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:54.126255 29143 consensus_queue.cc:1035] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 123, Time since last communication: 0.000s
I20240503 14:12:54.126610 28735 raft_consensus.cc:1261] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 FOLLOWER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240503 14:12:54.127351 29000 consensus_queue.cc:1035] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20240503 14:12:54.130839 29000 raft_consensus.cc:2932] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index 105 to 106, VOTER f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:54.134470 26868 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f2db6b6ac8bb4fffb18e0b65dea27b5c with cas_config_opid_index 105: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:54.135659 28734 raft_consensus.cc:2932] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 105 to 106, VOTER f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:54.144588 26874 catalog_manager.cc:5484] T f2db6b6ac8bb4fffb18e0b65dea27b5c P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: config changed from index 105 to 106, VOTER f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "305daa9a4d874d9db0e5378e5d27b617" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:54.148787 27282 raft_consensus.cc:1261] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240503 14:12:54.152256 27264 tablet_service.cc:1514] Processing DeleteTablet for tablet f2db6b6ac8bb4fffb18e0b65dea27b5c with delete_type TABLET_DATA_TOMBSTONED (TS f6c292faf79e49edabdbc7a310a30503 not found in new config with opid_index 106) from {username='slave'} at 127.0.0.1:41116
I20240503 14:12:54.161981 29183 tablet_replica.cc:330] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503: stopping tablet replica
I20240503 14:12:54.162292 29183 raft_consensus.cc:2229] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:54.165779 29183 raft_consensus.cc:2258] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:54.167623 28902 raft_consensus.cc:2932] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.126: config changed from index 124 to 126, VOTER 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) evicted. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:54.177561 26866 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4cca566ca06643ffbc72f02be6d544b4 with cas_config_opid_index 124: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:54.178378 29113 tablet_copy_client.cc:308] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.70:44045
I20240503 14:12:54.179452 26875 catalog_manager.cc:5484] T 4cca566ca06643ffbc72f02be6d544b4 P f6c292faf79e49edabdbc7a310a30503 reported cstate change: config changed from index 124 to 126, VOTER 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:54.187630 27132 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cca566ca06643ffbc72f02be6d544b4 with delete_type TABLET_DATA_TOMBSTONED (TS 305daa9a4d874d9db0e5378e5d27b617 not found in new config with opid_index 126) from {username='slave'} at 127.0.0.1:54420
I20240503 14:12:54.191479 28736 raft_consensus.cc:2932] T 4cca566ca06643ffbc72f02be6d544b4 P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 FOLLOWER]: Committing config change with OpId 1.126: config changed from index 124 to 126, VOTER 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) evicted. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:54.192404 29215 ts_tablet_manager.cc:1888] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.82
I20240503 14:12:54.192548 29215 log.cc:1196] T ab63a1d4c9764427a141feb6614fe18f P 820e18ddcc1e4d53821d43bc32652bc9: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/ab63a1d4c9764427a141feb6614fe18f
I20240503 14:12:54.193370 26866 catalog_manager.cc:4830] TS 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969): tablet ab63a1d4c9764427a141feb6614fe18f (table rebalance_test_table_2 [id=8c704c1747ee4a41b347496dc856e7bf]) successfully deleted
I20240503 14:12:54.198877 29183 ts_tablet_manager.cc:1875] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:54.239843 28625 tablet_copy_service.cc:140] P 89a56aa776934c4582bfbdb0622b0cb3: Received BeginTabletCopySession request for tablet 09ca4d63eab743b3977a47bcf120b3cf from peer 6b69a8c25e2b4de0bb10db8d7bdf83f0 ({username='slave'} at 127.22.242.71:47161)
I20240503 14:12:54.240135 28625 tablet_copy_service.cc:161] P 89a56aa776934c4582bfbdb0622b0cb3: Beginning new tablet copy session on tablet 09ca4d63eab743b3977a47bcf120b3cf from peer 6b69a8c25e2b4de0bb10db8d7bdf83f0 at {username='slave'} at 127.22.242.71:47161: session id = 6b69a8c25e2b4de0bb10db8d7bdf83f0-09ca4d63eab743b3977a47bcf120b3cf
I20240503 14:12:54.240728 29194 tablet_bootstrap.cc:492] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3: Bootstrap starting.
I20240503 14:12:54.260480 28625 tablet_copy_source_session.cc:205] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:54.262853 29113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ca4d63eab743b3977a47bcf120b3cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:54.289270 29113 tablet_copy_client.cc:783] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:54.289629 29113 tablet_copy_client.cc:647] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:54.291920 29113 tablet_copy_client.cc:523] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:54.303318 29113 tablet_bootstrap.cc:492] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Bootstrap starting.
I20240503 14:12:54.326428 29113 tablet_bootstrap.cc:492] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:54.327265 29113 tablet_bootstrap.cc:492] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Bootstrap complete.
I20240503 14:12:54.327687 29113 ts_tablet_manager.cc:1367] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.008s
I20240503 14:12:54.328629 29113 raft_consensus.cc:348] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } }
I20240503 14:12:54.329020 29113 raft_consensus.cc:729] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6b69a8c25e2b4de0bb10db8d7bdf83f0, State: Initialized, Role: LEARNER
I20240503 14:12:54.329275 29113 consensus_queue.cc:260] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } }
I20240503 14:12:54.330742 29113 ts_tablet_manager.cc:1398] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:12:54.331496 28625 tablet_copy_service.cc:342] P 89a56aa776934c4582bfbdb0622b0cb3: Request end of tablet copy session 6b69a8c25e2b4de0bb10db8d7bdf83f0-09ca4d63eab743b3977a47bcf120b3cf received from {username='slave'} at 127.22.242.71:47161
I20240503 14:12:54.331683 28625 tablet_copy_service.cc:433] P 89a56aa776934c4582bfbdb0622b0cb3: ending tablet copy session 6b69a8c25e2b4de0bb10db8d7bdf83f0-09ca4d63eab743b3977a47bcf120b3cf on tablet 09ca4d63eab743b3977a47bcf120b3cf with peer 6b69a8c25e2b4de0bb10db8d7bdf83f0
I20240503 14:12:54.342236 29216 ts_tablet_manager.cc:1888] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.125
I20240503 14:12:54.342522 29216 log.cc:1196] T c6d0c7755fb8449ea077c3786dc619fc P 305daa9a4d874d9db0e5378e5d27b617: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/c6d0c7755fb8449ea077c3786dc619fc
I20240503 14:12:54.343588 26868 catalog_manager.cc:4830] TS 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751): tablet c6d0c7755fb8449ea077c3786dc619fc (table rebalance_test_table_1 [id=34a5ee8103e54b178f47b78610c6a027]) successfully deleted
I20240503 14:12:54.344218 29216 tablet_replica.cc:330] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617: stopping tablet replica
I20240503 14:12:54.344516 29216 raft_consensus.cc:2229] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:54.345058 29216 pending_rounds.cc:62] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617: Trying to abort 1 pending ops.
I20240503 14:12:54.345191 29216 pending_rounds.cc:68] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617: Aborting op as it isn't in flight: id { term: 1 index: 125 } timestamp: 7023597871543586816 op_type: WRITE_OP write_request { tablet_id: "4cca566ca06643ffbc72f02be6d544b4" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\027d\314Uc\225\232:\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""24eb3c269f742d1a" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023597871363383296 authz_token { token_data: "\010\214\344\323\261\006\"3\n\005slave\022*\n da6fa437d89c4c1184580d93ab404751\020\001\030\001 \001(\001" signature: "<redacted>""U\220\316R\016\204\225\3765\334a\215?F\321/\037w\323\231\330\021\016,\030B\177f3_\302\006\335M\034\252\026\345\263\315\263\356E\314D\374\235\275n\"\356\354tg\204`r<`\263\261rS\032" signing_key_seq_num: 0 } } request_id { client_id: "e0fd98c895104304928df23a64f01fb7" seq_no: 342 first_incomplete_seq_no: 342 attempt_no: 0 }
I20240503 14:12:54.345885 29216 raft_consensus.cc:2258] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:54.346956 29216 ts_tablet_manager.cc:1875] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:54.375638 29183 ts_tablet_manager.cc:1888] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.104
I20240503 14:12:54.388875 29216 ts_tablet_manager.cc:1888] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.125
I20240503 14:12:54.389096 29216 log.cc:1196] T 4cca566ca06643ffbc72f02be6d544b4 P 305daa9a4d874d9db0e5378e5d27b617: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/4cca566ca06643ffbc72f02be6d544b4
I20240503 14:12:54.390267 26868 catalog_manager.cc:4830] TS 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751): tablet 4cca566ca06643ffbc72f02be6d544b4 (table rebalance_test_table_3 [id=da6fa437d89c4c1184580d93ab404751]) successfully deleted
I20240503 14:12:54.399096 29183 log.cc:1196] T f2db6b6ac8bb4fffb18e0b65dea27b5c P f6c292faf79e49edabdbc7a310a30503: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/f2db6b6ac8bb4fffb18e0b65dea27b5c
I20240503 14:12:54.403304 26866 catalog_manager.cc:4830] TS f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423): tablet f2db6b6ac8bb4fffb18e0b65dea27b5c (table rebalance_test_table_1 [id=34a5ee8103e54b178f47b78610c6a027]) successfully deleted
I20240503 14:12:54.473601 29063 tablet_bootstrap.cc:492] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=91 overwritten=0 applied=90 ignored=0} inserts{seen=721 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:54.474123 29063 tablet_bootstrap.cc:492] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3: Bootstrap complete.
I20240503 14:12:54.474434 29063 ts_tablet_manager.cc:1367] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3: Time spent bootstrapping tablet: real 1.122s user 0.168s sys 0.000s
I20240503 14:12:54.475298 29063 raft_consensus.cc:348] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } }
I20240503 14:12:54.476214 29063 raft_consensus.cc:729] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 89a56aa776934c4582bfbdb0622b0cb3, State: Initialized, Role: LEARNER
I20240503 14:12:54.476457 29063 consensus_queue.cc:260] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 90, Last appended: 1.91, Last appended by leader: 91, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } }
I20240503 14:12:54.477087 29063 ts_tablet_manager.cc:1398] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:54.483211 27304 tablet_copy_service.cc:342] P f6c292faf79e49edabdbc7a310a30503: Request end of tablet copy session 89a56aa776934c4582bfbdb0622b0cb3-efa780a04e854838a933be5934cd1578 received from {username='slave'} at 127.22.242.70:56359
I20240503 14:12:54.483350 27304 tablet_copy_service.cc:433] P f6c292faf79e49edabdbc7a310a30503: ending tablet copy session 89a56aa776934c4582bfbdb0622b0cb3-efa780a04e854838a933be5934cd1578 on tablet efa780a04e854838a933be5934cd1578 with peer 89a56aa776934c4582bfbdb0622b0cb3
W20240503 14:12:54.499102 29159 log.cc:924] Time spent T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20240503 14:12:54.566407 29173 tablet_bootstrap.cc:492] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516: Bootstrap replayed 1/1 log segments. Stats: ops{read=115 overwritten=0 applied=114 ignored=0} inserts{seen=799 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:54.567006 29173 tablet_bootstrap.cc:492] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516: Bootstrap complete.
I20240503 14:12:54.567335 29173 ts_tablet_manager.cc:1367] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516: Time spent bootstrapping tablet: real 0.503s user 0.175s sys 0.018s
I20240503 14:12:54.568148 29173 raft_consensus.cc:348] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } }
I20240503 14:12:54.568830 29173 raft_consensus.cc:729] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cbb4bf53151a4061a81cf5bacf084516, State: Initialized, Role: LEARNER
I20240503 14:12:54.569058 29173 consensus_queue.cc:260] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 114, Last appended: 1.115, Last appended by leader: 115, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } }
I20240503 14:12:54.679725 28736 raft_consensus.cc:1203] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
W20240503 14:12:54.802091 28538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.71:42765, user_credentials={real_user=slave}} blocked reactor thread for 120846us
I20240503 14:12:54.812938 28605 raft_consensus.cc:1203] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.82->[1.83-1.106] Dedup: 1.91->[1.92-1.106]
W20240503 14:12:54.850709 26958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.66:38751, user_credentials={real_user=slave}} blocked reactor thread for 48556.9us
I20240503 14:12:54.860467 29159 tablet_bootstrap.cc:492] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=73 overwritten=0 applied=73 ignored=0} inserts{seen=758 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:54.860985 29159 tablet_bootstrap.cc:492] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Bootstrap complete.
I20240503 14:12:54.861263 29159 ts_tablet_manager.cc:1367] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Time spent bootstrapping tablet: real 1.061s user 0.130s sys 0.017s
I20240503 14:12:54.862246 29159 raft_consensus.cc:348] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } }
I20240503 14:12:54.862489 29159 raft_consensus.cc:729] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6b69a8c25e2b4de0bb10db8d7bdf83f0, State: Initialized, Role: LEARNER
I20240503 14:12:54.862715 29159 consensus_queue.cc:260] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 73, Last appended: 1.73, Last appended by leader: 73, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: true } }
I20240503 14:12:54.866567 29159 ts_tablet_manager.cc:1398] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 14:12:54.871723 28736 raft_consensus.cc:1203] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.65->[1.66-1.82] Dedup: 1.73->[1.74-1.82]
I20240503 14:12:54.877732 27304 tablet_copy_service.cc:342] P f6c292faf79e49edabdbc7a310a30503: Request end of tablet copy session 6b69a8c25e2b4de0bb10db8d7bdf83f0-27288fd1dcc8421dab42d3c282f718ae received from {username='slave'} at 127.22.242.71:58253
I20240503 14:12:54.877882 27304 tablet_copy_service.cc:433] P f6c292faf79e49edabdbc7a310a30503: ending tablet copy session 6b69a8c25e2b4de0bb10db8d7bdf83f0-27288fd1dcc8421dab42d3c282f718ae on tablet 27288fd1dcc8421dab42d3c282f718ae with peer 6b69a8c25e2b4de0bb10db8d7bdf83f0
I20240503 14:12:54.929386 29173 ts_tablet_manager.cc:1398] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516: Time spent starting tablet: real 0.362s user 0.052s sys 0.102s
I20240503 14:12:54.933691 27174 tablet_copy_service.cc:342] P 305daa9a4d874d9db0e5378e5d27b617: Request end of tablet copy session cbb4bf53151a4061a81cf5bacf084516-f24cc60aba81429ca7275571faf77cb9 received from {username='slave'} at 127.22.242.69:60405
I20240503 14:12:54.933893 27174 tablet_copy_service.cc:433] P 305daa9a4d874d9db0e5378e5d27b617: ending tablet copy session cbb4bf53151a4061a81cf5bacf084516-f24cc60aba81429ca7275571faf77cb9 on tablet f24cc60aba81429ca7275571faf77cb9 with peer cbb4bf53151a4061a81cf5bacf084516
I20240503 14:12:55.012234 29146 raft_consensus.cc:1050] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503: attempting to promote NON_VOTER 6b69a8c25e2b4de0bb10db8d7bdf83f0 to VOTER
I20240503 14:12:55.013015 29146 consensus_queue.cc:237] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 1.88, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } }
I20240503 14:12:55.019268 28736 raft_consensus.cc:1261] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 LEARNER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 88. Preceding OpId from leader: term: 1 index: 89. (index mismatch)
I20240503 14:12:55.023499 28902 consensus_queue.cc:1035] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20240503 14:12:55.028268 29147 raft_consensus.cc:2932] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.89: config changed from index 66 to 89, 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:55.034973 28734 raft_consensus.cc:2932] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 FOLLOWER]: Committing config change with OpId 1.89: config changed from index 66 to 89, 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:55.042905 27021 raft_consensus.cc:1261] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 88. Preceding OpId from leader: term: 1 index: 89. (index mismatch)
I20240503 14:12:55.054392 28902 consensus_queue.cc:1035] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20240503 14:12:55.057292 27023 raft_consensus.cc:2932] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.89: config changed from index 66 to 89, 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:55.068508 29180 raft_consensus.cc:1050] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3: attempting to promote NON_VOTER 6b69a8c25e2b4de0bb10db8d7bdf83f0 to VOTER
I20240503 14:12:55.069440 29180 consensus_queue.cc:237] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } }
I20240503 14:12:55.085016 28734 raft_consensus.cc:1261] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 2 LEARNER]: Refusing update from remote peer 89a56aa776934c4582bfbdb0622b0cb3: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:12:55.085712 29243 consensus_queue.cc:1035] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:55.087658 27282 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "41b3bc788e164ce99c05a492ee768724"
dest_uuid: "f6c292faf79e49edabdbc7a310a30503"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59968
I20240503 14:12:55.087927 27282 raft_consensus.cc:595] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Received request to transfer leadership
W20240503 14:12:55.090066 27241 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60980: Service unavailable: leader transfer in progress
I20240503 14:12:55.093031 27151 raft_consensus.cc:1261] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Refusing update from remote peer 89a56aa776934c4582bfbdb0622b0cb3: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:12:55.093739 29180 consensus_queue.cc:1035] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:55.094959 28833 raft_consensus.cc:979] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503: : Instructing follower 4165ee79ee03466981e067d998339611 to start an election
I20240503 14:12:55.095135 28833 raft_consensus.cc:1067] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Signalling peer 4165ee79ee03466981e067d998339611 to start an election
I20240503 14:12:55.098781 29243 raft_consensus.cc:2932] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:55.104542 28735 raft_consensus.cc:2932] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:55.104651 26874 catalog_manager.cc:5484] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 reported cstate change: config changed from index 3 to 4, 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "89a56aa776934c4582bfbdb0622b0cb3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:55.108359 28340 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "41b3bc788e164ce99c05a492ee768724"
dest_uuid: "4165ee79ee03466981e067d998339611"
from {username='slave'} at 127.22.242.67:43533
I20240503 14:12:55.108704 28340 raft_consensus.cc:482] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:55.108901 28340 raft_consensus.cc:3037] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:55.111806 28340 raft_consensus.cc:504] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } }
W20240503 14:12:55.113165 27241 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60980: Service unavailable: leader transfer in progress
I20240503 14:12:55.115162 26875 catalog_manager.cc:5484] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9 reported cstate change: config changed from index 66 to 89, 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:55.115927 27282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b3bc788e164ce99c05a492ee768724" candidate_uuid: "4165ee79ee03466981e067d998339611" candidate_term: 2 candidate_status { last_received { term: 1 index: 95 } } ignore_live_leader: true dest_uuid: "f6c292faf79e49edabdbc7a310a30503"
I20240503 14:12:55.116185 27282 raft_consensus.cc:3032] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:55.116276 27282 raft_consensus.cc:729] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6c292faf79e49edabdbc7a310a30503, State: Running, Role: LEADER
I20240503 14:12:55.116613 27282 consensus_queue.cc:260] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.95, Last appended by leader: 95, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } }
I20240503 14:12:55.116983 27282 raft_consensus.cc:3037] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:55.118129 28340 leader_election.cc:288] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [CANDIDATE]: Term 2 election: Requested vote from peers f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423), 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
W20240503 14:12:55.128299 27241 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60980: Illegal state: replica f6c292faf79e49edabdbc7a310a30503 is not leader of this config: current role FOLLOWER
I20240503 14:12:55.133340 27282 raft_consensus.cc:2454] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4165ee79ee03466981e067d998339611 in term 2.
I20240503 14:12:55.134027 28274 leader_election.cc:302] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4165ee79ee03466981e067d998339611, f6c292faf79e49edabdbc7a310a30503; no voters:
W20240503 14:12:55.138945 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51154: Illegal state: replica 820e18ddcc1e4d53821d43bc32652bc9 is not leader of this config: current role FOLLOWER
I20240503 14:12:55.157867 27150 raft_consensus.cc:2932] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 6b69a8c25e2b4de0bb10db8d7bdf83f0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:55.173223 29279 raft_consensus.cc:2781] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:55.177343 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b3bc788e164ce99c05a492ee768724" candidate_uuid: "4165ee79ee03466981e067d998339611" candidate_term: 2 candidate_status { last_received { term: 1 index: 95 } } ignore_live_leader: true dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9"
I20240503 14:12:55.177623 27021 raft_consensus.cc:3037] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:55.181438 27021 raft_consensus.cc:2454] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4165ee79ee03466981e067d998339611 in term 2.
I20240503 14:12:55.230125 29194 tablet_bootstrap.cc:492] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=78 overwritten=0 applied=78 ignored=0} inserts{seen=709 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:55.230664 29194 tablet_bootstrap.cc:492] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3: Bootstrap complete.
I20240503 14:12:55.230962 29194 ts_tablet_manager.cc:1367] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3: Time spent bootstrapping tablet: real 0.990s user 0.165s sys 0.047s
I20240503 14:12:55.235014 29194 raft_consensus.cc:348] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } }
I20240503 14:12:55.235291 29194 raft_consensus.cc:729] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 89a56aa776934c4582bfbdb0622b0cb3, State: Initialized, Role: LEARNER
I20240503 14:12:55.235523 29194 consensus_queue.cc:260] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 78, Last appended: 1.78, Last appended by leader: 78, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } }
I20240503 14:12:55.238385 28471 raft_consensus.cc:1203] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.104->[1.105-1.132] Dedup: 1.115->[1.116-1.132]
I20240503 14:12:55.240254 29194 ts_tablet_manager.cc:1398] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240503 14:12:55.241078 27043 tablet_copy_service.cc:342] P 820e18ddcc1e4d53821d43bc32652bc9: Request end of tablet copy session 89a56aa776934c4582bfbdb0622b0cb3-7efea07ad3204b6e868a05e9d80f5b3a received from {username='slave'} at 127.22.242.70:58447
I20240503 14:12:55.241227 27043 tablet_copy_service.cc:433] P 820e18ddcc1e4d53821d43bc32652bc9: ending tablet copy session 89a56aa776934c4582bfbdb0622b0cb3-7efea07ad3204b6e868a05e9d80f5b3a on tablet 7efea07ad3204b6e868a05e9d80f5b3a with peer 89a56aa776934c4582bfbdb0622b0cb3
I20240503 14:12:55.285140 29146 raft_consensus.cc:1050] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503: attempting to promote NON_VOTER 89a56aa776934c4582bfbdb0622b0cb3 to VOTER
I20240503 14:12:55.286120 28605 raft_consensus.cc:1203] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.71->[1.72-1.95] Dedup: 1.78->[1.79-1.95]
I20240503 14:12:55.286011 29146 consensus_queue.cc:237] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } }
I20240503 14:12:55.287928 28604 raft_consensus.cc:1261] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 LEARNER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch)
I20240503 14:12:55.288427 27150 raft_consensus.cc:1261] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch)
I20240503 14:12:55.289170 28902 consensus_queue.cc:1035] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20240503 14:12:55.289696 28902 consensus_queue.cc:1035] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20240503 14:12:55.292616 29147 raft_consensus.cc:2932] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index 83 to 107, 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:12:55.296193 28604 raft_consensus.cc:2932] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 83 to 107, 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:12:55.308701 27151 raft_consensus.cc:2932] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 83 to 107, 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
W20240503 14:12:55.331064 27242 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60980: Illegal state: replica f6c292faf79e49edabdbc7a310a30503 is not leader of this config: current role FOLLOWER
W20240503 14:12:55.342247 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51154: Illegal state: replica 820e18ddcc1e4d53821d43bc32652bc9 is not leader of this config: current role FOLLOWER
I20240503 14:12:55.373826 28605 consensus_queue.cc:237] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } }
I20240503 14:12:55.376109 28734 raft_consensus.cc:1261] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 2 FOLLOWER]: Refusing update from remote peer 89a56aa776934c4582bfbdb0622b0cb3: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:12:55.376699 29242 consensus_queue.cc:1035] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:55.379129 29242 raft_consensus.cc:2932] T 09ca4d63eab743b3977a47bcf120b3cf P 89a56aa776934c4582bfbdb0622b0cb3 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:55.383342 28734 raft_consensus.cc:2932] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:55.393620 28963 raft_consensus.cc:1050] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9: attempting to promote NON_VOTER 89a56aa776934c4582bfbdb0622b0cb3 to VOTER
I20240503 14:12:55.394537 28963 consensus_queue.cc:237] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.96, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } }
I20240503 14:12:55.397647 27281 raft_consensus.cc:1261] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240503 14:12:55.398326 28235 consensus_queue.cc:1035] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 95, Time since last communication: 0.000s
I20240503 14:12:55.400489 26868 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 09ca4d63eab743b3977a47bcf120b3cf with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:55.402738 29171 raft_consensus.cc:2932] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Committing config change with OpId 1.97: config changed from index 72 to 97, 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:12:55.410913 27281 raft_consensus.cc:2932] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 72 to 97, 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:12:55.423568 28605 raft_consensus.cc:1261] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 LEARNER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240503 14:12:55.448745 28882 consensus_queue.cc:1035] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 95, Time since last communication: 0.000s
I20240503 14:12:55.452157 28603 raft_consensus.cc:2932] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 72 to 97, 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:12:55.453756 27282 raft_consensus.cc:1203] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.95->[1.96-1.96] Dedup: 1.96->[]
I20240503 14:12:55.475397 27282 consensus_queue.cc:237] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 1.97, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } }
I20240503 14:12:55.478044 28734 raft_consensus.cc:1261] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240503 14:12:55.480743 29146 consensus_queue.cc:1035] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20240503 14:12:55.483304 29147 raft_consensus.cc:2932] T 27288fd1dcc8421dab42d3c282f718ae P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.98: config changed from index 89 to 98, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:55.487275 26866 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 27288fd1dcc8421dab42d3c282f718ae with cas_config_opid_index 89: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:55.489753 28734 raft_consensus.cc:2932] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0 [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index 89 to 98, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:55.517275 29279 raft_consensus.cc:686] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [term 2 LEADER]: Becoming Leader. State: Replica: 4165ee79ee03466981e067d998339611, State: Running, Role: LEADER
I20240503 14:12:55.518151 29279 consensus_queue.cc:237] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.95, Last appended by leader: 95, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } }
I20240503 14:12:55.527108 28854 raft_consensus.cc:1050] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617: attempting to promote NON_VOTER cbb4bf53151a4061a81cf5bacf084516 to VOTER
I20240503 14:12:55.528198 28854 consensus_queue.cc:237] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 135, Committed index: 135, Last appended: 1.135, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } }
I20240503 14:12:55.534168 28472 raft_consensus.cc:1261] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [term 1 LEARNER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 1 index: 135. Preceding OpId from leader: term: 1 index: 136. (index mismatch)
I20240503 14:12:55.541637 26876 catalog_manager.cc:5484] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 reported cstate change: config changed from index 83 to 107, 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:12:55.545434 27282 raft_consensus.cc:1261] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 1 index: 135. Preceding OpId from leader: term: 1 index: 136. (index mismatch)
I20240503 14:12:55.545989 28999 consensus_queue.cc:1035] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 135, Time since last communication: 0.000s
I20240503 14:12:55.546991 26882 catalog_manager.cc:5484] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 reported cstate change: term changed from 1 to 2, leader changed from f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) to 4165ee79ee03466981e067d998339611 (127.22.242.68). New cstate: current_term: 2 leader_uuid: "4165ee79ee03466981e067d998339611" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:55.553195 29000 raft_consensus.cc:2932] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 LEADER]: Committing config change with OpId 1.136: config changed from index 105 to 136, cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:55.555152 27283 raft_consensus.cc:2932] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Committing config change with OpId 1.136: config changed from index 105 to 136, cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:55.559871 26877 catalog_manager.cc:5484] T 09ca4d63eab743b3977a47bcf120b3cf P 6b69a8c25e2b4de0bb10db8d7bdf83f0 reported cstate change: config changed from index 4 to 5, VOTER 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) evicted. New cstate: current_term: 2 leader_uuid: "89a56aa776934c4582bfbdb0622b0cb3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:55.560994 28854 consensus_queue.cc:1035] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 135, Time since last communication: 0.000s
I20240503 14:12:55.561137 26874 catalog_manager.cc:5484] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 reported cstate change: config changed from index 72 to 97, 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:55.564680 28471 raft_consensus.cc:2932] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [term 1 FOLLOWER]: Committing config change with OpId 1.136: config changed from index 105 to 136, cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
W20240503 14:12:55.567203 26868 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 09ca4d63eab743b3977a47bcf120b3cf with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70:44045) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 4 but the committed config has opid_index of 5
I20240503 14:12:55.575482 27132 tablet_service.cc:1514] Processing DeleteTablet for tablet 09ca4d63eab743b3977a47bcf120b3cf with delete_type TABLET_DATA_TOMBSTONED (TS 305daa9a4d874d9db0e5378e5d27b617 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54420
I20240503 14:12:55.576706 27021 raft_consensus.cc:1261] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Refusing update from remote peer 4165ee79ee03466981e067d998339611: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 2 index: 97. (index mismatch)
I20240503 14:12:55.578299 29284 consensus_queue.cc:1035] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240503 14:12:55.579355 29302 tablet_replica.cc:330] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617: stopping tablet replica
I20240503 14:12:55.580862 26876 catalog_manager.cc:5484] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 reported cstate change: config changed from index 105 to 136, cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "305daa9a4d874d9db0e5378e5d27b617" committed_config { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:55.583878 29302 raft_consensus.cc:2229] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:55.584236 29302 raft_consensus.cc:2258] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:55.591228 26877 catalog_manager.cc:5484] T 27288fd1dcc8421dab42d3c282f718ae P 6b69a8c25e2b4de0bb10db8d7bdf83f0 reported cstate change: config changed from index 89 to 98, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "6b69a8c25e2b4de0bb10db8d7bdf83f0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 42765 } attrs { promote: false } } }
I20240503 14:12:55.592991 29302 ts_tablet_manager.cc:1875] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:55.596335 27282 consensus_queue.cc:237] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 1.117, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } }
I20240503 14:12:55.597920 27003 tablet_service.cc:1514] Processing DeleteTablet for tablet 27288fd1dcc8421dab42d3c282f718ae with delete_type TABLET_DATA_TOMBSTONED (TS 820e18ddcc1e4d53821d43bc32652bc9 not found in new config with opid_index 98) from {username='slave'} at 127.0.0.1:40428
I20240503 14:12:55.600000 27021 consensus_queue.cc:237] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 1.101, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } }
I20240503 14:12:55.600876 29266 mvcc.cc:205] Tried to move back new op lower bound from 7023597877552472064 to 7023597877329018880. Current Snapshot: MvccSnapshot[applied={T|T < 7023597877552472064}]
I20240503 14:12:55.603269 27282 raft_consensus.cc:1261] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 2 FOLLOWER]: Refusing update from remote peer 4165ee79ee03466981e067d998339611: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 2 index: 97. (index mismatch)
W20240503 14:12:55.603962 26866 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 27288fd1dcc8421dab42d3c282f718ae with cas_config_opid_index 89: ChangeConfig:REMOVE_PEER failed with leader f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 89 but the committed config has opid_index of 98
I20240503 14:12:55.604420 28603 raft_consensus.cc:1261] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240503 14:12:55.605363 28603 raft_consensus.cc:1261] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240503 14:12:55.606169 29303 consensus_queue.cc:1035] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240503 14:12:55.606660 29170 consensus_queue.cc:1035] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20240503 14:12:55.607199 28902 consensus_queue.cc:1035] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20240503 14:12:55.616290 29170 raft_consensus.cc:2932] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Committing config change with OpId 1.102: config changed from index 97 to 102, VOTER f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) evicted. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:12:55.618820 28902 raft_consensus.cc:2932] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.118: config changed from index 107 to 118, VOTER 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) evicted. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:12:55.622653 28603 raft_consensus.cc:2932] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 97 to 102, VOTER f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) evicted. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:12:55.629700 28605 raft_consensus.cc:2932] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index 107 to 118, VOTER 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) evicted. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:12:55.633242 29302 ts_tablet_manager.cc:1888] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 14:12:55.633392 29302 log.cc:1196] T 09ca4d63eab743b3977a47bcf120b3cf P 305daa9a4d874d9db0e5378e5d27b617: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/09ca4d63eab743b3977a47bcf120b3cf
I20240503 14:12:55.639192 26866 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet efa780a04e854838a933be5934cd1578 with cas_config_opid_index 107: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:55.641690 26875 catalog_manager.cc:5484] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 reported cstate change: config changed from index 107 to 118, VOTER 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:55.643554 29304 tablet_replica.cc:330] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9: stopping tablet replica
I20240503 14:12:55.644400 26868 catalog_manager.cc:4830] TS 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751): tablet 09ca4d63eab743b3977a47bcf120b3cf (table TestTable [id=fd924edbfa2840c2bca0f99e7e0778a7]) successfully deleted
I20240503 14:12:55.645287 29304 raft_consensus.cc:2229] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:55.647159 26866 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7efea07ad3204b6e868a05e9d80f5b3a with cas_config_opid_index 97: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:55.648224 29304 raft_consensus.cc:2258] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:55.649288 26877 catalog_manager.cc:5484] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3 reported cstate change: config changed from index 97 to 102, VOTER f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:12:55.649699 29304 ts_tablet_manager.cc:1875] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:55.659035 27132 tablet_service.cc:1514] Processing DeleteTablet for tablet efa780a04e854838a933be5934cd1578 with delete_type TABLET_DATA_TOMBSTONED (TS 305daa9a4d874d9db0e5378e5d27b617 not found in new config with opid_index 118) from {username='slave'} at 127.0.0.1:54420
I20240503 14:12:55.659303 27264 tablet_service.cc:1514] Processing DeleteTablet for tablet 7efea07ad3204b6e868a05e9d80f5b3a with delete_type TABLET_DATA_TOMBSTONED (TS f6c292faf79e49edabdbc7a310a30503 not found in new config with opid_index 102) from {username='slave'} at 127.0.0.1:41116
I20240503 14:12:55.659441 29302 tablet_replica.cc:330] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617: stopping tablet replica
I20240503 14:12:55.659622 29302 raft_consensus.cc:2229] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:55.659837 29302 raft_consensus.cc:2258] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:55.661633 29302 ts_tablet_manager.cc:1875] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:55.664201 29304 ts_tablet_manager.cc:1888] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.97
I20240503 14:12:55.669477 29304 log.cc:1196] T 27288fd1dcc8421dab42d3c282f718ae P 820e18ddcc1e4d53821d43bc32652bc9: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/27288fd1dcc8421dab42d3c282f718ae
I20240503 14:12:55.670567 26866 catalog_manager.cc:4830] TS 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969): tablet 27288fd1dcc8421dab42d3c282f718ae (table rebalance_test_table_0 [id=eda67e22fc6146289f3862c2470418fb]) successfully deleted
I20240503 14:12:55.676874 29302 ts_tablet_manager.cc:1888] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.117
I20240503 14:12:55.677162 29302 log.cc:1196] T efa780a04e854838a933be5934cd1578 P 305daa9a4d874d9db0e5378e5d27b617: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/efa780a04e854838a933be5934cd1578
I20240503 14:12:55.680330 26868 catalog_manager.cc:4830] TS 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751): tablet efa780a04e854838a933be5934cd1578 (table rebalance_test_table_1 [id=34a5ee8103e54b178f47b78610c6a027]) successfully deleted
I20240503 14:12:55.687459 29309 tablet_replica.cc:330] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503: stopping tablet replica
I20240503 14:12:55.687810 29309 raft_consensus.cc:2229] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:55.697983 29309 raft_consensus.cc:2258] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:55.702980 27151 consensus_queue.cc:237] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 137, Committed index: 137, Last appended: 1.137, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } }
I20240503 14:12:55.707744 28340 consensus_queue.cc:237] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 2.98, Last appended by leader: 95, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } }
I20240503 14:12:55.710093 27023 raft_consensus.cc:1261] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Refusing update from remote peer 4165ee79ee03466981e067d998339611: Log matching property violated. Preceding OpId in replica: term: 2 index: 98. Preceding OpId from leader: term: 2 index: 99. (index mismatch)
I20240503 14:12:55.711151 29303 consensus_queue.cc:1035] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240503 14:12:55.714140 29284 raft_consensus.cc:2932] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [term 2 LEADER]: Committing config change with OpId 2.99: config changed from index 80 to 99, VOTER f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) evicted. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:12:55.714745 28471 raft_consensus.cc:1261] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [term 1 FOLLOWER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 1 index: 137. Preceding OpId from leader: term: 1 index: 138. (index mismatch)
I20240503 14:12:55.719033 26866 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 41b3bc788e164ce99c05a492ee768724 with cas_config_opid_index 80: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:55.720992 26877 catalog_manager.cc:5484] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 reported cstate change: config changed from index 80 to 99, VOTER f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) evicted. New cstate: current_term: 2 leader_uuid: "4165ee79ee03466981e067d998339611" committed_config { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:55.721480 27023 raft_consensus.cc:2932] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Committing config change with OpId 2.99: config changed from index 80 to 99, VOTER f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) evicted. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:12:55.726454 28999 consensus_queue.cc:1035] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 138, Last known committed idx: 136, Time since last communication: 0.000s
I20240503 14:12:55.727182 29309 ts_tablet_manager.cc:1875] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:55.740780 27264 tablet_service.cc:1514] Processing DeleteTablet for tablet 41b3bc788e164ce99c05a492ee768724 with delete_type TABLET_DATA_TOMBSTONED (TS f6c292faf79e49edabdbc7a310a30503 not found in new config with opid_index 99) from {username='slave'} at 127.0.0.1:41116
I20240503 14:12:55.756328 29307 raft_consensus.cc:2932] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 LEADER]: Committing config change with OpId 1.138: config changed from index 136 to 138, VOTER f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) evicted. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:55.773202 26868 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f24cc60aba81429ca7275571faf77cb9 with cas_config_opid_index 136: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:55.781500 29309 ts_tablet_manager.cc:1888] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.101
I20240503 14:12:55.781654 29309 log.cc:1196] T 7efea07ad3204b6e868a05e9d80f5b3a P f6c292faf79e49edabdbc7a310a30503: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/7efea07ad3204b6e868a05e9d80f5b3a
I20240503 14:12:55.782460 26866 catalog_manager.cc:4830] TS f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423): tablet 7efea07ad3204b6e868a05e9d80f5b3a (table rebalance_test_table_0 [id=eda67e22fc6146289f3862c2470418fb]) successfully deleted
I20240503 14:12:55.783324 29309 tablet_replica.cc:330] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503: stopping tablet replica
I20240503 14:12:55.783530 29309 raft_consensus.cc:2229] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:55.783798 29309 raft_consensus.cc:2258] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:55.784040 28472 raft_consensus.cc:2932] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [term 1 FOLLOWER]: Committing config change with OpId 1.138: config changed from index 136 to 138, VOTER f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) evicted. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:55.785163 26877 catalog_manager.cc:5484] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: config changed from index 136 to 138, VOTER f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "305daa9a4d874d9db0e5378e5d27b617" committed_config { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:55.788692 29309 ts_tablet_manager.cc:1875] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:55.789868 27264 tablet_service.cc:1514] Processing DeleteTablet for tablet f24cc60aba81429ca7275571faf77cb9 with delete_type TABLET_DATA_TOMBSTONED (TS f6c292faf79e49edabdbc7a310a30503 not found in new config with opid_index 138) from {username='slave'} at 127.0.0.1:41116
I20240503 14:12:55.797631 29309 ts_tablet_manager.cc:1888] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.98
I20240503 14:12:55.797797 29309 log.cc:1196] T 41b3bc788e164ce99c05a492ee768724 P f6c292faf79e49edabdbc7a310a30503: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/41b3bc788e164ce99c05a492ee768724
I20240503 14:12:55.798512 29309 tablet_replica.cc:330] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503: stopping tablet replica
I20240503 14:12:55.798714 29309 raft_consensus.cc:2229] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:55.798941 29309 raft_consensus.cc:2258] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:55.799067 26866 catalog_manager.cc:4830] TS f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423): tablet 41b3bc788e164ce99c05a492ee768724 (table rebalance_test_table_4 [id=6cdd58518c114d93a32a80b11512af8f]) successfully deleted
I20240503 14:12:55.800722 29309 ts_tablet_manager.cc:1875] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:55.816045 29309 ts_tablet_manager.cc:1888] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.137
I20240503 14:12:55.816335 29309 log.cc:1196] T f24cc60aba81429ca7275571faf77cb9 P f6c292faf79e49edabdbc7a310a30503: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/f24cc60aba81429ca7275571faf77cb9
I20240503 14:12:55.817631 26866 catalog_manager.cc:4830] TS f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423): tablet f24cc60aba81429ca7275571faf77cb9 (table rebalance_test_table_3 [id=da6fa437d89c4c1184580d93ab404751]) successfully deleted
I20240503 14:12:57.473403 27132 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:57.583022 28321 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:57.697321 28716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:57.714430 28452 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:57.716600 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:57.719341 27003 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:57.763743 27264 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:58.103247 27021 consensus_queue.cc:237] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 167, Committed index: 167, Last appended: 1.167, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } }
I20240503 14:12:58.106552 27150 raft_consensus.cc:1261] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240503 14:12:58.107513 29434 consensus_queue.cc:1035] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
W20240503 14:12:58.112670 26958 consensus_peers.cc:485] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 -> Peer cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69:39081): Couldn't send request to peer cbb4bf53151a4061a81cf5bacf084516. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 927839713c994a9d9ad9f071a6943b0d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:58.115299 29434 raft_consensus.cc:2932] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Committing config change with OpId 1.168: config changed from index -1 to 168, NON_VOTER cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) added. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } } }
I20240503 14:12:58.116402 27150 raft_consensus.cc:2932] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index -1 to 168, NON_VOTER cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) added. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } } }
I20240503 14:12:58.123448 26875 catalog_manager.cc:5484] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: config changed from index -1 to 168, NON_VOTER cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } } }
I20240503 14:12:58.294548 28340 consensus_queue.cc:237] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 147, Committed index: 147, Last appended: 2.148, Last appended by leader: 95, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } }
I20240503 14:12:58.310812 27023 raft_consensus.cc:1261] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Refusing update from remote peer 4165ee79ee03466981e067d998339611: Log matching property violated. Preceding OpId in replica: term: 2 index: 148. Preceding OpId from leader: term: 2 index: 149. (index mismatch)
I20240503 14:12:58.313197 29361 consensus_queue.cc:1035] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 149, Last known committed idx: 147, Time since last communication: 0.000s
I20240503 14:12:58.316473 29284 raft_consensus.cc:2932] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [term 2 LEADER]: Committing config change with OpId 2.149: config changed from index 99 to 149, NON_VOTER 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) added. New config: { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } } }
I20240503 14:12:58.322333 26877 catalog_manager.cc:5484] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 reported cstate change: config changed from index 99 to 149, NON_VOTER 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) added. New cstate: current_term: 2 leader_uuid: "4165ee79ee03466981e067d998339611" committed_config { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:58.325474 27023 raft_consensus.cc:2932] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Committing config change with OpId 2.149: config changed from index 99 to 149, NON_VOTER 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) added. New config: { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } } }
W20240503 14:12:58.336318 28276 consensus_peers.cc:485] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 -> Peer 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70:44045): Couldn't send request to peer 89a56aa776934c4582bfbdb0622b0cb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41b3bc788e164ce99c05a492ee768724. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:58.438210 27151 consensus_queue.cc:237] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 193, Committed index: 193, Last appended: 1.193, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 194 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } }
I20240503 14:12:58.453085 28471 raft_consensus.cc:1261] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [term 1 FOLLOWER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 1 index: 193. Preceding OpId from leader: term: 1 index: 194. (index mismatch)
I20240503 14:12:58.454443 28999 consensus_queue.cc:1035] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 194, Last known committed idx: 193, Time since last communication: 0.000s
I20240503 14:12:58.464192 28999 raft_consensus.cc:2932] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 LEADER]: Committing config change with OpId 1.194: config changed from index 138 to 194, NON_VOTER 4165ee79ee03466981e067d998339611 (127.22.242.68) added. New config: { opid_index: 194 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } } }
I20240503 14:12:58.465829 28471 raft_consensus.cc:2932] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [term 1 FOLLOWER]: Committing config change with OpId 1.194: config changed from index 138 to 194, NON_VOTER 4165ee79ee03466981e067d998339611 (127.22.242.68) added. New config: { opid_index: 194 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } } }
I20240503 14:12:58.480249 26877 catalog_manager.cc:5484] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: config changed from index 138 to 194, NON_VOTER 4165ee79ee03466981e067d998339611 (127.22.242.68) added. New cstate: current_term: 1 leader_uuid: "305daa9a4d874d9db0e5378e5d27b617" committed_config { opid_index: 194 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:58.506821 29473 ts_tablet_manager.cc:902] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516: Initiating tablet copy from peer 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
W20240503 14:12:58.511090 27087 consensus_peers.cc:485] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 -> Peer 4165ee79ee03466981e067d998339611 (127.22.242.68:41205): Couldn't send request to peer 4165ee79ee03466981e067d998339611. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f24cc60aba81429ca7275571faf77cb9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:58.563956 27021 consensus_queue.cc:237] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 156, Committed index: 156, Last appended: 1.156, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } }
I20240503 14:12:58.567214 27151 raft_consensus.cc:1261] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 157. (index mismatch)
I20240503 14:12:58.568068 29391 consensus_queue.cc:1035] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 157, Last known committed idx: 156, Time since last communication: 0.000s
I20240503 14:12:58.594748 29434 raft_consensus.cc:2932] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Committing config change with OpId 1.157: config changed from index -1 to 157, NON_VOTER 4165ee79ee03466981e067d998339611 (127.22.242.68) added. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } } }
W20240503 14:12:58.596365 26956 consensus_peers.cc:485] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 -> Peer 4165ee79ee03466981e067d998339611 (127.22.242.68:41205): Couldn't send request to peer 4165ee79ee03466981e067d998339611. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eee65dca344f41528758a1babd90a2ff. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:58.602659 27151 raft_consensus.cc:2932] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Committing config change with OpId 1.157: config changed from index -1 to 157, NON_VOTER 4165ee79ee03466981e067d998339611 (127.22.242.68) added. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } } }
I20240503 14:12:58.606856 26875 catalog_manager.cc:5484] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 reported cstate change: config changed from index -1 to 157, NON_VOTER 4165ee79ee03466981e067d998339611 (127.22.242.68) added. New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:58.653193 29473 tablet_copy_client.cc:308] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:37969
I20240503 14:12:58.654311 27043 tablet_copy_service.cc:140] P 820e18ddcc1e4d53821d43bc32652bc9: Received BeginTabletCopySession request for tablet 927839713c994a9d9ad9f071a6943b0d from peer cbb4bf53151a4061a81cf5bacf084516 ({username='slave'} at 127.22.242.69:55871)
I20240503 14:12:58.654502 27043 tablet_copy_service.cc:161] P 820e18ddcc1e4d53821d43bc32652bc9: Beginning new tablet copy session on tablet 927839713c994a9d9ad9f071a6943b0d from peer cbb4bf53151a4061a81cf5bacf084516 at {username='slave'} at 127.22.242.69:55871: session id = cbb4bf53151a4061a81cf5bacf084516-927839713c994a9d9ad9f071a6943b0d
I20240503 14:12:58.657280 27043 tablet_copy_source_session.cc:205] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:58.658756 29473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927839713c994a9d9ad9f071a6943b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:58.668624 29473 tablet_copy_client.cc:783] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:58.668903 29473 tablet_copy_client.cc:647] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:58.682536 29473 tablet_copy_client.cc:523] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:58.686506 29473 tablet_bootstrap.cc:492] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516: Bootstrap starting.
I20240503 14:12:58.860078 29494 ts_tablet_manager.cc:902] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3: Initiating tablet copy from peer 4165ee79ee03466981e067d998339611 (127.22.242.68:41205)
I20240503 14:12:58.999801 29494 tablet_copy_client.cc:308] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.68:41205
I20240503 14:12:59.070081 29508 ts_tablet_manager.cc:902] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611: Initiating tablet copy from peer 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
I20240503 14:12:59.159817 29504 ts_tablet_manager.cc:902] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611: Initiating tablet copy from peer 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:12:59.193244 28361 tablet_copy_service.cc:140] P 4165ee79ee03466981e067d998339611: Received BeginTabletCopySession request for tablet 41b3bc788e164ce99c05a492ee768724 from peer 89a56aa776934c4582bfbdb0622b0cb3 ({username='slave'} at 127.22.242.70:35409)
I20240503 14:12:59.193490 28361 tablet_copy_service.cc:161] P 4165ee79ee03466981e067d998339611: Beginning new tablet copy session on tablet 41b3bc788e164ce99c05a492ee768724 from peer 89a56aa776934c4582bfbdb0622b0cb3 at {username='slave'} at 127.22.242.70:35409: session id = 89a56aa776934c4582bfbdb0622b0cb3-41b3bc788e164ce99c05a492ee768724
I20240503 14:12:59.198401 28361 tablet_copy_source_session.cc:205] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:59.200415 29494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b3bc788e164ce99c05a492ee768724. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.206645 29494 tablet_copy_client.cc:783] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:59.206923 29494 tablet_copy_client.cc:647] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:59.209575 29494 tablet_copy_client.cc:523] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:59.213047 29494 tablet_bootstrap.cc:492] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3: Bootstrap starting.
I20240503 14:12:59.306877 29473 tablet_bootstrap.cc:492] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516: Bootstrap replayed 1/1 log segments. Stats: ops{read=178 overwritten=0 applied=178 ignored=0} inserts{seen=850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:59.307642 29473 tablet_bootstrap.cc:492] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516: Bootstrap complete.
I20240503 14:12:59.308112 29473 ts_tablet_manager.cc:1367] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516: Time spent bootstrapping tablet: real 0.622s user 0.232s sys 0.037s
I20240503 14:12:59.309011 29473 raft_consensus.cc:348] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } }
I20240503 14:12:59.310810 29473 raft_consensus.cc:729] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cbb4bf53151a4061a81cf5bacf084516, State: Initialized, Role: LEARNER
I20240503 14:12:59.311257 29473 consensus_queue.cc:260] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 178, Last appended: 1.178, Last appended by leader: 178, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: true } }
I20240503 14:12:59.391410 29473 ts_tablet_manager.cc:1398] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516: Time spent starting tablet: real 0.021s user 0.005s sys 0.002s
I20240503 14:12:59.394989 27043 tablet_copy_service.cc:342] P 820e18ddcc1e4d53821d43bc32652bc9: Request end of tablet copy session cbb4bf53151a4061a81cf5bacf084516-927839713c994a9d9ad9f071a6943b0d received from {username='slave'} at 127.22.242.69:55871
I20240503 14:12:59.395176 27043 tablet_copy_service.cc:433] P 820e18ddcc1e4d53821d43bc32652bc9: ending tablet copy session cbb4bf53151a4061a81cf5bacf084516-927839713c994a9d9ad9f071a6943b0d on tablet 927839713c994a9d9ad9f071a6943b0d with peer cbb4bf53151a4061a81cf5bacf084516
I20240503 14:12:59.421270 28470 raft_consensus.cc:1203] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.167->[1.168-1.186] Dedup: 1.178->[1.179-1.186]
I20240503 14:12:59.519690 29504 tablet_copy_client.cc:308] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:38751
I20240503 14:12:59.580024 29391 raft_consensus.cc:1050] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9: attempting to promote NON_VOTER cbb4bf53151a4061a81cf5bacf084516 to VOTER
I20240503 14:12:59.580193 29508 tablet_copy_client.cc:308] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:37969
I20240503 14:12:59.580780 29391 consensus_queue.cc:237] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 188, Committed index: 188, Last appended: 1.189, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 190 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } }
I20240503 14:12:59.586858 27043 tablet_copy_service.cc:140] P 820e18ddcc1e4d53821d43bc32652bc9: Received BeginTabletCopySession request for tablet eee65dca344f41528758a1babd90a2ff from peer 4165ee79ee03466981e067d998339611 ({username='slave'} at 127.22.242.68:44031)
I20240503 14:12:59.587044 27043 tablet_copy_service.cc:161] P 820e18ddcc1e4d53821d43bc32652bc9: Beginning new tablet copy session on tablet eee65dca344f41528758a1babd90a2ff from peer 4165ee79ee03466981e067d998339611 at {username='slave'} at 127.22.242.68:44031: session id = 4165ee79ee03466981e067d998339611-eee65dca344f41528758a1babd90a2ff
I20240503 14:12:59.588214 27151 raft_consensus.cc:1261] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 189. Preceding OpId from leader: term: 1 index: 190. (index mismatch)
I20240503 14:12:59.589458 28963 consensus_queue.cc:1035] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 190, Last known committed idx: 188, Time since last communication: 0.000s
I20240503 14:12:59.589888 27043 tablet_copy_source_session.cc:205] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:59.590368 27174 tablet_copy_service.cc:140] P 305daa9a4d874d9db0e5378e5d27b617: Received BeginTabletCopySession request for tablet f24cc60aba81429ca7275571faf77cb9 from peer 4165ee79ee03466981e067d998339611 ({username='slave'} at 127.22.242.68:36737)
I20240503 14:12:59.590519 27174 tablet_copy_service.cc:161] P 305daa9a4d874d9db0e5378e5d27b617: Beginning new tablet copy session on tablet f24cc60aba81429ca7275571faf77cb9 from peer 4165ee79ee03466981e067d998339611 at {username='slave'} at 127.22.242.68:36737: session id = 4165ee79ee03466981e067d998339611-f24cc60aba81429ca7275571faf77cb9
I20240503 14:12:59.593297 27174 tablet_copy_source_session.cc:205] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:59.595680 29508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee65dca344f41528758a1babd90a2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.597057 28472 raft_consensus.cc:1261] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 1 LEARNER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 189. Preceding OpId from leader: term: 1 index: 190. (index mismatch)
I20240503 14:12:59.598131 29434 consensus_queue.cc:1035] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 190, Last known committed idx: 188, Time since last communication: 0.000s
I20240503 14:12:59.604365 28472 raft_consensus.cc:2932] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 1 FOLLOWER]: Committing config change with OpId 1.190: config changed from index 168 to 190, cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 190 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:59.608227 29434 raft_consensus.cc:2932] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Committing config change with OpId 1.190: config changed from index 168 to 190, cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 190 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:59.617302 29504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f24cc60aba81429ca7275571faf77cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.626219 26875 catalog_manager.cc:5484] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 reported cstate change: config changed from index 168 to 190, cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: 190 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:59.632500 27150 raft_consensus.cc:2932] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Committing config change with OpId 1.190: config changed from index 168 to 190, cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 190 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:12:59.646291 29508 tablet_copy_client.cc:783] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:59.646660 29508 tablet_copy_client.cc:647] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:59.654621 29504 tablet_copy_client.cc:783] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:59.655529 29504 tablet_copy_client.cc:647] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:59.658368 29504 tablet_copy_client.cc:523] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:59.664393 29504 tablet_bootstrap.cc:492] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611: Bootstrap starting.
I20240503 14:12:59.749450 29508 tablet_copy_client.cc:523] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:59.755080 29508 tablet_bootstrap.cc:492] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611: Bootstrap starting.
W20240503 14:12:59.888613 26958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.70:44045, user_credentials={real_user=slave}} blocked reactor thread for 57849.4us
I20240503 14:12:59.915271 29494 tablet_bootstrap.cc:492] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=162 overwritten=0 applied=161 ignored=0} inserts{seen=842 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:59.915886 29494 tablet_bootstrap.cc:492] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3: Bootstrap complete.
I20240503 14:12:59.916272 29494 ts_tablet_manager.cc:1367] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3: Time spent bootstrapping tablet: real 0.703s user 0.282s sys 0.065s
I20240503 14:12:59.917101 29494 raft_consensus.cc:348] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3 [term 2 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } }
I20240503 14:12:59.917788 29494 raft_consensus.cc:729] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 89a56aa776934c4582bfbdb0622b0cb3, State: Initialized, Role: LEARNER
I20240503 14:12:59.918057 29494 consensus_queue.cc:260] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 161, Last appended: 2.162, Last appended by leader: 162, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: true } }
I20240503 14:12:59.919121 29494 ts_tablet_manager.cc:1398] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:12:59.921952 28361 tablet_copy_service.cc:342] P 4165ee79ee03466981e067d998339611: Request end of tablet copy session 89a56aa776934c4582bfbdb0622b0cb3-41b3bc788e164ce99c05a492ee768724 received from {username='slave'} at 127.22.242.70:35409
I20240503 14:12:59.922130 28361 tablet_copy_service.cc:433] P 4165ee79ee03466981e067d998339611: ending tablet copy session 89a56aa776934c4582bfbdb0622b0cb3-41b3bc788e164ce99c05a492ee768724 on tablet 41b3bc788e164ce99c05a492ee768724 with peer 89a56aa776934c4582bfbdb0622b0cb3
W20240503 14:12:59.950455 26958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.70:44045, user_credentials={real_user=slave}} blocked reactor thread for 45589us
W20240503 14:13:00.082672 29508 log.cc:924] Time spent T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611: Append to log took a long time: real 0.059s user 0.002s sys 0.000s
I20240503 14:13:00.256894 28604 raft_consensus.cc:1203] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.148->[2.149-2.171] Dedup: 2.162->[2.163-2.171]
I20240503 14:13:00.278889 29508 tablet_bootstrap.cc:492] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611: Bootstrap replayed 1/1 log segments. Stats: ops{read=176 overwritten=0 applied=176 ignored=0} inserts{seen=848 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:00.279525 29508 tablet_bootstrap.cc:492] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611: Bootstrap complete.
I20240503 14:13:00.279924 29508 ts_tablet_manager.cc:1367] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611: Time spent bootstrapping tablet: real 0.525s user 0.235s sys 0.025s
I20240503 14:13:00.281106 29508 raft_consensus.cc:348] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } }
I20240503 14:13:00.281502 29508 raft_consensus.cc:729] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4165ee79ee03466981e067d998339611, State: Initialized, Role: LEARNER
I20240503 14:13:00.281736 29508 consensus_queue.cc:260] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 176, Last appended: 1.176, Last appended by leader: 176, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } }
I20240503 14:13:00.282974 29508 ts_tablet_manager.cc:1398] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:00.300863 27043 tablet_copy_service.cc:342] P 820e18ddcc1e4d53821d43bc32652bc9: Request end of tablet copy session 4165ee79ee03466981e067d998339611-eee65dca344f41528758a1babd90a2ff received from {username='slave'} at 127.22.242.68:44031
I20240503 14:13:00.301046 27043 tablet_copy_service.cc:433] P 820e18ddcc1e4d53821d43bc32652bc9: ending tablet copy session 4165ee79ee03466981e067d998339611-eee65dca344f41528758a1babd90a2ff on tablet eee65dca344f41528758a1babd90a2ff with peer 4165ee79ee03466981e067d998339611
I20240503 14:13:00.507743 28340 raft_consensus.cc:1203] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.156->[1.157-1.187] Dedup: 1.176->[1.177-1.187]
I20240503 14:13:00.532562 27021 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "927839713c994a9d9ad9f071a6943b0d"
dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53408
I20240503 14:13:00.532855 27021 raft_consensus.cc:595] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:00.536715 29504 tablet_bootstrap.cc:492] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611: Bootstrap replayed 1/1 log segments. Stats: ops{read=222 overwritten=0 applied=222 ignored=0} inserts{seen=904 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:00.537382 29504 tablet_bootstrap.cc:492] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611: Bootstrap complete.
I20240503 14:13:00.537899 29504 ts_tablet_manager.cc:1367] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611: Time spent bootstrapping tablet: real 0.874s user 0.297s sys 0.025s
I20240503 14:13:00.539119 29504 raft_consensus.cc:348] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 194 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } }
I20240503 14:13:00.539620 29504 raft_consensus.cc:729] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4165ee79ee03466981e067d998339611, State: Initialized, Role: LEARNER
I20240503 14:13:00.539948 29504 consensus_queue.cc:260] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 222, Last appended: 1.222, Last appended by leader: 222, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 194 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: true } }
I20240503 14:13:00.540894 29504 ts_tablet_manager.cc:1398] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:00.541357 29552 raft_consensus.cc:979] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9: : Instructing follower 305daa9a4d874d9db0e5378e5d27b617 to start an election
I20240503 14:13:00.541523 29552 raft_consensus.cc:1067] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Signalling peer 305daa9a4d874d9db0e5378e5d27b617 to start an election
I20240503 14:13:00.542688 27174 tablet_copy_service.cc:342] P 305daa9a4d874d9db0e5378e5d27b617: Request end of tablet copy session 4165ee79ee03466981e067d998339611-f24cc60aba81429ca7275571faf77cb9 received from {username='slave'} at 127.22.242.68:36737
I20240503 14:13:00.542673 27154 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "927839713c994a9d9ad9f071a6943b0d"
dest_uuid: "305daa9a4d874d9db0e5378e5d27b617"
from {username='slave'} at 127.22.242.65:54731
I20240503 14:13:00.542876 27174 tablet_copy_service.cc:433] P 305daa9a4d874d9db0e5378e5d27b617: ending tablet copy session 4165ee79ee03466981e067d998339611-f24cc60aba81429ca7275571faf77cb9 on tablet f24cc60aba81429ca7275571faf77cb9 with peer 4165ee79ee03466981e067d998339611
I20240503 14:13:00.542968 27154 raft_consensus.cc:482] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:00.543097 27154 raft_consensus.cc:3037] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:00.546764 27154 raft_consensus.cc:504] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 190 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } }
I20240503 14:13:00.548674 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927839713c994a9d9ad9f071a6943b0d" candidate_uuid: "305daa9a4d874d9db0e5378e5d27b617" candidate_term: 2 candidate_status { last_received { term: 1 index: 204 } } ignore_live_leader: true dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9"
I20240503 14:13:00.548794 28472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927839713c994a9d9ad9f071a6943b0d" candidate_uuid: "305daa9a4d874d9db0e5378e5d27b617" candidate_term: 2 candidate_status { last_received { term: 1 index: 204 } } ignore_live_leader: true dest_uuid: "cbb4bf53151a4061a81cf5bacf084516"
I20240503 14:13:00.548964 27021 raft_consensus.cc:3032] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:00.549082 27021 raft_consensus.cc:729] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Running, Role: LEADER
I20240503 14:13:00.549126 28472 raft_consensus.cc:3037] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:00.549460 27021 consensus_queue.cc:260] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 204, Committed index: 204, Last appended: 1.204, Last appended by leader: 204, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 190 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } }
I20240503 14:13:00.551781 27021 raft_consensus.cc:3037] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:00.552134 28472 raft_consensus.cc:2454] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 305daa9a4d874d9db0e5378e5d27b617 in term 2.
I20240503 14:13:00.552778 27089 leader_election.cc:302] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, cbb4bf53151a4061a81cf5bacf084516; no voters:
I20240503 14:13:00.553190 29456 raft_consensus.cc:2781] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:00.553339 29456 raft_consensus.cc:686] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 2 LEADER]: Becoming Leader. State: Replica: 305daa9a4d874d9db0e5378e5d27b617, State: Running, Role: LEADER
I20240503 14:13:00.553661 29456 consensus_queue.cc:237] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 203, Committed index: 203, Last appended: 1.204, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 190 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } }
I20240503 14:13:00.553865 27154 leader_election.cc:288] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 2 election: Requested vote from peers 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969), cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69:39081)
I20240503 14:13:00.558544 27021 raft_consensus.cc:2454] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 305daa9a4d874d9db0e5378e5d27b617 in term 2.
I20240503 14:13:00.558518 26873 catalog_manager.cc:5484] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: term changed from 1 to 2, leader changed from 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) to 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66). New cstate: current_term: 2 leader_uuid: "305daa9a4d874d9db0e5378e5d27b617" committed_config { opid_index: 190 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:00.582614 29553 raft_consensus.cc:1050] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9: attempting to promote NON_VOTER 4165ee79ee03466981e067d998339611 to VOTER
I20240503 14:13:00.583442 29553 consensus_queue.cc:237] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 188, Committed index: 188, Last appended: 1.188, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } }
I20240503 14:13:00.590023 27154 raft_consensus.cc:1261] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 188. Preceding OpId from leader: term: 1 index: 189. (index mismatch)
I20240503 14:13:00.590222 28340 raft_consensus.cc:1261] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [term 1 LEARNER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 188. Preceding OpId from leader: term: 1 index: 189. (index mismatch)
I20240503 14:13:00.590790 29434 consensus_queue.cc:1035] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 189, Last known committed idx: 188, Time since last communication: 0.000s
I20240503 14:13:00.591135 29391 consensus_queue.cc:1035] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 189, Last known committed idx: 188, Time since last communication: 0.000s
I20240503 14:13:00.594965 29434 raft_consensus.cc:2932] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Committing config change with OpId 1.189: config changed from index 157 to 189, 4165ee79ee03466981e067d998339611 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:13:00.601495 27154 raft_consensus.cc:2932] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Committing config change with OpId 1.189: config changed from index 157 to 189, 4165ee79ee03466981e067d998339611 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:13:00.601783 26875 catalog_manager.cc:5484] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 reported cstate change: config changed from index 157 to 189, 4165ee79ee03466981e067d998339611 (127.22.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:00.602491 28340 raft_consensus.cc:2932] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [term 1 FOLLOWER]: Committing config change with OpId 1.189: config changed from index 157 to 189, 4165ee79ee03466981e067d998339611 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:13:00.677004 29597 raft_consensus.cc:1050] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611: attempting to promote NON_VOTER 89a56aa776934c4582bfbdb0622b0cb3 to VOTER
I20240503 14:13:00.677973 29597 consensus_queue.cc:237] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 171, Committed index: 171, Last appended: 2.171, Last appended by leader: 95, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } }
I20240503 14:13:00.692241 28603 raft_consensus.cc:1261] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3 [term 2 LEARNER]: Refusing update from remote peer 4165ee79ee03466981e067d998339611: Log matching property violated. Preceding OpId in replica: term: 2 index: 171. Preceding OpId from leader: term: 2 index: 172. (index mismatch)
I20240503 14:13:00.693872 29597 consensus_queue.cc:1035] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 172, Last known committed idx: 171, Time since last communication: 0.000s
I20240503 14:13:00.697288 27021 raft_consensus.cc:1261] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Refusing update from remote peer 4165ee79ee03466981e067d998339611: Log matching property violated. Preceding OpId in replica: term: 2 index: 171. Preceding OpId from leader: term: 2 index: 172. (index mismatch)
I20240503 14:13:00.697813 28603 raft_consensus.cc:2932] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3 [term 2 FOLLOWER]: Committing config change with OpId 2.172: config changed from index 149 to 172, 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:13:00.698527 29600 consensus_queue.cc:1035] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 172, Last known committed idx: 171, Time since last communication: 0.001s
I20240503 14:13:00.700613 27021 raft_consensus.cc:2932] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Committing config change with OpId 2.172: config changed from index 149 to 172, 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:13:00.705178 26877 catalog_manager.cc:5484] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3 reported cstate change: config changed from index 149 to 172, 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "4165ee79ee03466981e067d998339611" committed_config { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:13:00.706100 29542 raft_consensus.cc:2932] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [term 2 LEADER]: Committing config change with OpId 2.172: config changed from index 149 to 172, 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:13:00.713076 28340 consensus_queue.cc:237] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 172, Committed index: 172, Last appended: 2.172, Last appended by leader: 95, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 173 OBSOLETE_local: false peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } }
I20240503 14:13:00.715090 28603 raft_consensus.cc:1261] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3 [term 2 FOLLOWER]: Refusing update from remote peer 4165ee79ee03466981e067d998339611: Log matching property violated. Preceding OpId in replica: term: 2 index: 172. Preceding OpId from leader: term: 2 index: 173. (index mismatch)
I20240503 14:13:00.718537 29542 consensus_queue.cc:1035] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 173, Last known committed idx: 172, Time since last communication: 0.000s
I20240503 14:13:00.721910 29600 raft_consensus.cc:2932] T 41b3bc788e164ce99c05a492ee768724 P 4165ee79ee03466981e067d998339611 [term 2 LEADER]: Committing config change with OpId 2.173: config changed from index 172 to 173, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New config: { opid_index: 173 OBSOLETE_local: false peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:13:00.726155 26866 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 41b3bc788e164ce99c05a492ee768724 with cas_config_opid_index 172: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:00.728439 28603 raft_consensus.cc:2932] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3 [term 2 FOLLOWER]: Committing config change with OpId 2.173: config changed from index 172 to 173, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New config: { opid_index: 173 OBSOLETE_local: false peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:13:00.746048 26877 catalog_manager.cc:5484] T 41b3bc788e164ce99c05a492ee768724 P 89a56aa776934c4582bfbdb0622b0cb3 reported cstate change: config changed from index 172 to 173, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New cstate: current_term: 2 leader_uuid: "4165ee79ee03466981e067d998339611" committed_config { opid_index: 173 OBSOLETE_local: false peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } }
I20240503 14:13:00.753881 27003 tablet_service.cc:1514] Processing DeleteTablet for tablet 41b3bc788e164ce99c05a492ee768724 with delete_type TABLET_DATA_TOMBSTONED (TS 820e18ddcc1e4d53821d43bc32652bc9 not found in new config with opid_index 173) from {username='slave'} at 127.0.0.1:40428
I20240503 14:13:00.776942 29602 tablet_replica.cc:330] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9: stopping tablet replica
I20240503 14:13:00.777344 29602 raft_consensus.cc:2229] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:00.777817 29602 raft_consensus.cc:2258] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:00.780908 29602 ts_tablet_manager.cc:1875] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:00.853615 29602 ts_tablet_manager.cc:1888] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.172
I20240503 14:13:00.853956 29602 log.cc:1196] T 41b3bc788e164ce99c05a492ee768724 P 820e18ddcc1e4d53821d43bc32652bc9: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/41b3bc788e164ce99c05a492ee768724
I20240503 14:13:00.855916 26866 catalog_manager.cc:4830] TS 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969): tablet 41b3bc788e164ce99c05a492ee768724 (table rebalance_test_table_4 [id=6cdd58518c114d93a32a80b11512af8f]) successfully deleted
I20240503 14:13:00.947651 28340 raft_consensus.cc:1203] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.193->[1.194-1.256] Dedup: 1.222->[1.223-1.256]
I20240503 14:13:00.972342 27021 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eee65dca344f41528758a1babd90a2ff"
dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53418
I20240503 14:13:00.972558 27021 raft_consensus.cc:595] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:00.979846 28471 raft_consensus.cc:1261] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 2 FOLLOWER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 1 index: 204. Preceding OpId from leader: term: 2 index: 205. (index mismatch)
I20240503 14:13:00.980518 28854 consensus_queue.cc:1035] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 205, Last known committed idx: 203, Time since last communication: 0.000s
W20240503 14:13:01.044406 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
W20240503 14:13:01.048211 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
I20240503 14:13:01.048487 27021 raft_consensus.cc:1261] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 1 index: 204. Preceding OpId from leader: term: 2 index: 205. (index mismatch)
I20240503 14:13:01.049296 28854 consensus_queue.cc:1035] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 205, Last known committed idx: 204, Time since last communication: 0.000s
W20240503 14:13:01.053906 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
W20240503 14:13:01.055415 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51152: Illegal state: replica 820e18ddcc1e4d53821d43bc32652bc9 is not leader of this config: current role FOLLOWER
I20240503 14:13:01.058571 27150 consensus_queue.cc:237] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 2.205, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } }
W20240503 14:13:01.059139 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
I20240503 14:13:01.063308 28471 raft_consensus.cc:1261] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 2 FOLLOWER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 2 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20240503 14:13:01.064275 29591 consensus_queue.cc:1035] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 203, Time since last communication: 0.000s
W20240503 14:13:01.067559 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
W20240503 14:13:01.075541 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
I20240503 14:13:01.076092 29456 raft_consensus.cc:2932] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 2 LEADER]: Committing config change with OpId 2.206: config changed from index 190 to 206, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
I20240503 14:13:01.080276 26868 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 927839713c994a9d9ad9f071a6943b0d with cas_config_opid_index 190: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:01.083030 28472 raft_consensus.cc:2932] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 2 FOLLOWER]: Committing config change with OpId 2.206: config changed from index 190 to 206, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } }
W20240503 14:13:01.085443 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
I20240503 14:13:01.092679 26877 catalog_manager.cc:5484] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: config changed from index 190 to 206, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New cstate: current_term: 2 leader_uuid: "305daa9a4d874d9db0e5378e5d27b617" committed_config { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:13:01.094198 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
I20240503 14:13:01.101087 27003 tablet_service.cc:1514] Processing DeleteTablet for tablet 927839713c994a9d9ad9f071a6943b0d with delete_type TABLET_DATA_TOMBSTONED (TS 820e18ddcc1e4d53821d43bc32652bc9 not found in new config with opid_index 206) from {username='slave'} at 127.0.0.1:40428
I20240503 14:13:01.101532 29602 tablet_replica.cc:330] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9: stopping tablet replica
I20240503 14:13:01.101759 29602 raft_consensus.cc:2229] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:01.101977 29602 raft_consensus.cc:2258] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:01.103706 29602 ts_tablet_manager.cc:1875] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 14:13:01.106127 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
I20240503 14:13:01.107367 29456 raft_consensus.cc:1050] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617: attempting to promote NON_VOTER 4165ee79ee03466981e067d998339611 to VOTER
I20240503 14:13:01.108083 29456 consensus_queue.cc:237] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 263, Committed index: 263, Last appended: 1.263, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } }
I20240503 14:13:01.114575 28470 raft_consensus.cc:1261] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [term 1 FOLLOWER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 1 index: 263. Preceding OpId from leader: term: 1 index: 264. (index mismatch)
W20240503 14:13:01.117362 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
I20240503 14:13:01.119941 29528 consensus_queue.cc:1035] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20240503 14:13:01.124404 28340 raft_consensus.cc:1261] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611 [term 1 LEARNER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 1 index: 263. Preceding OpId from leader: term: 1 index: 264. (index mismatch)
I20240503 14:13:01.125490 29528 consensus_queue.cc:1035] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20240503 14:13:01.126749 29591 raft_consensus.cc:2932] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 LEADER]: Committing config change with OpId 1.264: config changed from index 194 to 264, 4165ee79ee03466981e067d998339611 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:13:01.128612 29602 ts_tablet_manager.cc:1888] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.205
I20240503 14:13:01.128768 29602 log.cc:1196] T 927839713c994a9d9ad9f071a6943b0d P 820e18ddcc1e4d53821d43bc32652bc9: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/927839713c994a9d9ad9f071a6943b0d
I20240503 14:13:01.129796 26866 catalog_manager.cc:4830] TS 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969): tablet 927839713c994a9d9ad9f071a6943b0d (table rebalance_test_table_2 [id=8c704c1747ee4a41b347496dc856e7bf]) successfully deleted
I20240503 14:13:01.131136 28471 raft_consensus.cc:2932] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [term 1 FOLLOWER]: Committing config change with OpId 1.264: config changed from index 194 to 264, 4165ee79ee03466981e067d998339611 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
W20240503 14:13:01.132495 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
I20240503 14:13:01.140717 26877 catalog_manager.cc:5484] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: config changed from index 194 to 264, 4165ee79ee03466981e067d998339611 (127.22.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "305daa9a4d874d9db0e5378e5d27b617" committed_config { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:01.143476 28340 raft_consensus.cc:2932] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611 [term 1 FOLLOWER]: Committing config change with OpId 1.264: config changed from index 194 to 264, 4165ee79ee03466981e067d998339611 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
W20240503 14:13:01.145915 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
W20240503 14:13:01.163584 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
W20240503 14:13:01.180491 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
W20240503 14:13:01.196442 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
I20240503 14:13:01.196413 27154 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f24cc60aba81429ca7275571faf77cb9"
dest_uuid: "305daa9a4d874d9db0e5378e5d27b617"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43300
I20240503 14:13:01.196729 27154 raft_consensus.cc:595] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:01.208408 28854 raft_consensus.cc:979] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617: : Instructing follower cbb4bf53151a4061a81cf5bacf084516 to start an election
I20240503 14:13:01.208580 28854 raft_consensus.cc:1067] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 LEADER]: Signalling peer cbb4bf53151a4061a81cf5bacf084516 to start an election
I20240503 14:13:01.210150 28470 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f24cc60aba81429ca7275571faf77cb9"
dest_uuid: "cbb4bf53151a4061a81cf5bacf084516"
from {username='slave'} at 127.22.242.66:42507
I20240503 14:13:01.211550 28470 raft_consensus.cc:482] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:01.211797 28470 raft_consensus.cc:3037] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:01.217695 28470 raft_consensus.cc:504] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } }
I20240503 14:13:01.220537 27154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f24cc60aba81429ca7275571faf77cb9" candidate_uuid: "cbb4bf53151a4061a81cf5bacf084516" candidate_term: 2 candidate_status { last_received { term: 1 index: 267 } } ignore_live_leader: true dest_uuid: "305daa9a4d874d9db0e5378e5d27b617"
I20240503 14:13:01.220841 27154 raft_consensus.cc:3032] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 LEADER]: Stepping down as leader of term 1
W20240503 14:13:01.220899 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
I20240503 14:13:01.220937 27154 raft_consensus.cc:729] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 305daa9a4d874d9db0e5378e5d27b617, State: Running, Role: LEADER
I20240503 14:13:01.221284 27154 consensus_queue.cc:260] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 1.267, Last appended by leader: 267, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } }
I20240503 14:13:01.221758 27154 raft_consensus.cc:3037] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:01.229907 27154 raft_consensus.cc:2454] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbb4bf53151a4061a81cf5bacf084516 in term 2.
I20240503 14:13:01.230774 28407 leader_election.cc:302] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, cbb4bf53151a4061a81cf5bacf084516; no voters:
I20240503 14:13:01.231302 29625 raft_consensus.cc:2781] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:01.231472 29625 raft_consensus.cc:686] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [term 2 LEADER]: Becoming Leader. State: Replica: cbb4bf53151a4061a81cf5bacf084516, State: Running, Role: LEADER
I20240503 14:13:01.232010 29625 consensus_queue.cc:237] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 1.267, Last appended by leader: 267, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } }
I20240503 14:13:01.238569 26873 catalog_manager.cc:5484] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 reported cstate change: term changed from 1 to 2, leader changed from 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) to cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69). New cstate: current_term: 2 leader_uuid: "cbb4bf53151a4061a81cf5bacf084516" committed_config { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 14:13:01.241344 26980 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
I20240503 14:13:01.250010 28470 leader_election.cc:288] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [CANDIDATE]: Term 2 election: Requested vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751), 4165ee79ee03466981e067d998339611 (127.22.242.68:41205)
W20240503 14:13:01.262166 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
W20240503 14:13:01.262651 27112 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52736: Illegal state: replica 305daa9a4d874d9db0e5378e5d27b617 is not leader of this config: current role FOLLOWER
W20240503 14:13:01.286307 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
W20240503 14:13:01.311592 26980 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
I20240503 14:13:01.320928 28340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f24cc60aba81429ca7275571faf77cb9" candidate_uuid: "cbb4bf53151a4061a81cf5bacf084516" candidate_term: 2 candidate_status { last_received { term: 1 index: 267 } } ignore_live_leader: true dest_uuid: "4165ee79ee03466981e067d998339611"
I20240503 14:13:01.321266 28340 raft_consensus.cc:3037] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:01.324214 28340 raft_consensus.cc:2454] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbb4bf53151a4061a81cf5bacf084516 in term 2.
W20240503 14:13:01.336342 26980 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
W20240503 14:13:01.361778 26980 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
I20240503 14:13:01.378572 27154 raft_consensus.cc:1261] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Refusing update from remote peer cbb4bf53151a4061a81cf5bacf084516: Log matching property violated. Preceding OpId in replica: term: 1 index: 267. Preceding OpId from leader: term: 2 index: 269. (index mismatch)
I20240503 14:13:01.378960 28340 raft_consensus.cc:1261] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611 [term 2 FOLLOWER]: Refusing update from remote peer cbb4bf53151a4061a81cf5bacf084516: Log matching property violated. Preceding OpId in replica: term: 1 index: 267. Preceding OpId from leader: term: 2 index: 269. (index mismatch)
I20240503 14:13:01.379863 29625 consensus_queue.cc:1035] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20240503 14:13:01.385241 29625 consensus_queue.cc:1035] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20240503 14:13:01.389456 29601 mvcc.cc:205] Tried to move back new op lower bound from 7023597901317619712 to 7023597900732547072. Current Snapshot: MvccSnapshot[applied={T|T < 7023597901317619712 or (T in {7023597901317619712})}]
I20240503 14:13:01.396252 29552 raft_consensus.cc:979] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9: : Instructing follower 4165ee79ee03466981e067d998339611 to start an election
I20240503 14:13:01.396420 29552 raft_consensus.cc:1067] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Signalling peer 4165ee79ee03466981e067d998339611 to start an election
I20240503 14:13:01.397326 28339 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eee65dca344f41528758a1babd90a2ff"
dest_uuid: "4165ee79ee03466981e067d998339611"
from {username='slave'} at 127.22.242.65:51009
I20240503 14:13:01.397572 28339 raft_consensus.cc:482] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:01.397681 28339 raft_consensus.cc:3037] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:01.420518 29563 mvcc.cc:205] Tried to move back new op lower bound from 7023597901317619712 to 7023597900732547072. Current Snapshot: MvccSnapshot[applied={T|T < 7023597901317619712}]
W20240503 14:13:01.455426 26980 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
W20240503 14:13:01.486192 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
I20240503 14:13:01.503827 28471 consensus_queue.cc:237] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 2.273, Last appended by leader: 267, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } }
I20240503 14:13:01.506621 28340 raft_consensus.cc:1261] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611 [term 2 FOLLOWER]: Refusing update from remote peer cbb4bf53151a4061a81cf5bacf084516: Log matching property violated. Preceding OpId in replica: term: 2 index: 273. Preceding OpId from leader: term: 2 index: 274. (index mismatch)
I20240503 14:13:01.507175 29650 consensus_queue.cc:1035] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 274, Last known committed idx: 272, Time since last communication: 0.000s
I20240503 14:13:01.509821 29650 raft_consensus.cc:2932] T f24cc60aba81429ca7275571faf77cb9 P cbb4bf53151a4061a81cf5bacf084516 [term 2 LEADER]: Committing config change with OpId 2.274: config changed from index 264 to 274, VOTER 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) evicted. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:13:01.514302 28340 raft_consensus.cc:2932] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611 [term 2 FOLLOWER]: Committing config change with OpId 2.274: config changed from index 264 to 274, VOTER 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) evicted. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
W20240503 14:13:01.516659 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
I20240503 14:13:01.524554 26868 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f24cc60aba81429ca7275571faf77cb9 with cas_config_opid_index 264: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:01.525064 26875 catalog_manager.cc:5484] T f24cc60aba81429ca7275571faf77cb9 P 4165ee79ee03466981e067d998339611 reported cstate change: config changed from index 264 to 274, VOTER 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) evicted. New cstate: current_term: 2 leader_uuid: "cbb4bf53151a4061a81cf5bacf084516" committed_config { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:13:01.528635 28339 raft_consensus.cc:504] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } }
I20240503 14:13:01.544530 27132 tablet_service.cc:1514] Processing DeleteTablet for tablet f24cc60aba81429ca7275571faf77cb9 with delete_type TABLET_DATA_TOMBSTONED (TS 305daa9a4d874d9db0e5378e5d27b617 not found in new config with opid_index 274) from {username='slave'} at 127.0.0.1:54420
W20240503 14:13:01.550079 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
I20240503 14:13:01.565356 27154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee65dca344f41528758a1babd90a2ff" candidate_uuid: "4165ee79ee03466981e067d998339611" candidate_term: 2 candidate_status { last_received { term: 1 index: 200 } } ignore_live_leader: true dest_uuid: "305daa9a4d874d9db0e5378e5d27b617"
I20240503 14:13:01.565608 27154 raft_consensus.cc:3037] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:01.568526 28339 leader_election.cc:288] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [CANDIDATE]: Term 2 election: Requested vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751), 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
I20240503 14:13:01.570434 27154 raft_consensus.cc:2454] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4165ee79ee03466981e067d998339611 in term 2.
I20240503 14:13:01.571051 28276 leader_election.cc:302] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, 4165ee79ee03466981e067d998339611; no voters:
I20240503 14:13:01.571372 29542 raft_consensus.cc:2781] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:01.571516 29542 raft_consensus.cc:686] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [term 2 LEADER]: Becoming Leader. State: Replica: 4165ee79ee03466981e067d998339611, State: Running, Role: LEADER
I20240503 14:13:01.571842 29542 consensus_queue.cc:237] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 200, Committed index: 200, Last appended: 1.200, Last appended by leader: 200, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } }
I20240503 14:13:01.575964 26875 catalog_manager.cc:5484] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 reported cstate change: term changed from 1 to 2, leader changed from 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) to 4165ee79ee03466981e067d998339611 (127.22.242.68). New cstate: current_term: 2 leader_uuid: "4165ee79ee03466981e067d998339611" committed_config { opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:13:01.580655 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Service unavailable: leader transfer in progress
I20240503 14:13:01.584475 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee65dca344f41528758a1babd90a2ff" candidate_uuid: "4165ee79ee03466981e067d998339611" candidate_term: 2 candidate_status { last_received { term: 1 index: 200 } } ignore_live_leader: true dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9"
I20240503 14:13:01.584717 27021 raft_consensus.cc:3032] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:01.584798 27021 raft_consensus.cc:729] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Running, Role: LEADER
I20240503 14:13:01.585064 27021 consensus_queue.cc:260] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 200, Committed index: 200, Last appended: 1.200, Last appended by leader: 200, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } }
I20240503 14:13:01.585423 27021 raft_consensus.cc:3037] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:01.589769 27021 raft_consensus.cc:2454] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4165ee79ee03466981e067d998339611 in term 2.
I20240503 14:13:01.593961 29657 tablet_replica.cc:330] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617: stopping tablet replica
I20240503 14:13:01.594453 29657 raft_consensus.cc:2229] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:01.600296 29657 pending_rounds.cc:62] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617: Trying to abort 1 pending ops.
I20240503 14:13:01.600896 29657 pending_rounds.cc:68] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617: Aborting op as it isn't in flight: id { term: 2 index: 270 } timestamp: 7023597901693841408 op_type: WRITE_OP write_request { tablet_id: "f24cc60aba81429ca7275571faf77cb9" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000oi\322\014\\*rR\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""d7697d5e9d4ba64f" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023597901666529280 authz_token { token_data: "\010\214\344\323\261\006\"3\n\005slave\022*\n da6fa437d89c4c1184580d93ab404751\020\001\030\001 \001(\001" signature: "<redacted>""U\220\316R\016\204\225\3765\334a\215?F\321/\037w\323\231\330\021\016,\030B\177f3_\302\006\335M\034\252\026\345\263\315\263\356E\314D\374\235\275n\"\356\354tg\204`r<`\263\261rS\032" signing_key_seq_num: 0 } } request_id { client_id: "e0fd98c895104304928df23a64f01fb7" seq_no: 794 first_incomplete_seq_no: 794 attempt_no: 0 }
I20240503 14:13:01.602330 29657 raft_consensus.cc:2258] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:01.603392 29657 ts_tablet_manager.cc:1875] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 14:13:01.614651 26980 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51138: Illegal state: replica 820e18ddcc1e4d53821d43bc32652bc9 is not leader of this config: current role FOLLOWER
I20240503 14:13:01.625442 29657 ts_tablet_manager.cc:1888] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.270
I20240503 14:13:01.625697 29657 log.cc:1196] T f24cc60aba81429ca7275571faf77cb9 P 305daa9a4d874d9db0e5378e5d27b617: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/f24cc60aba81429ca7275571faf77cb9
I20240503 14:13:01.626730 26868 catalog_manager.cc:4830] TS 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751): tablet f24cc60aba81429ca7275571faf77cb9 (table rebalance_test_table_3 [id=da6fa437d89c4c1184580d93ab404751]) successfully deleted
W20240503 14:13:01.738250 27113 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43328: Illegal state: replica 305daa9a4d874d9db0e5378e5d27b617 is not leader of this config: current role FOLLOWER
I20240503 14:13:01.798898 27023 raft_consensus.cc:1261] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Refusing update from remote peer 4165ee79ee03466981e067d998339611: Log matching property violated. Preceding OpId in replica: term: 1 index: 200. Preceding OpId from leader: term: 2 index: 202. (index mismatch)
I20240503 14:13:01.806211 29542 consensus_queue.cc:1035] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 201, Last known committed idx: 200, Time since last communication: 0.000s
I20240503 14:13:01.811946 27154 raft_consensus.cc:1261] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Refusing update from remote peer 4165ee79ee03466981e067d998339611: Log matching property violated. Preceding OpId in replica: term: 1 index: 200. Preceding OpId from leader: term: 2 index: 202. (index mismatch)
I20240503 14:13:01.815358 29599 consensus_queue.cc:1035] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 201, Last known committed idx: 200, Time since last communication: 0.000s
I20240503 14:13:01.847966 28341 consensus_queue.cc:237] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 203, Committed index: 203, Last appended: 2.203, Last appended by leader: 200, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } }
I20240503 14:13:01.852609 27154 raft_consensus.cc:1261] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Refusing update from remote peer 4165ee79ee03466981e067d998339611: Log matching property violated. Preceding OpId in replica: term: 2 index: 203. Preceding OpId from leader: term: 2 index: 204. (index mismatch)
I20240503 14:13:01.854797 29678 consensus_queue.cc:1035] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 204, Last known committed idx: 203, Time since last communication: 0.000s
I20240503 14:13:01.865126 29674 raft_consensus.cc:2932] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 [term 2 LEADER]: Committing config change with OpId 2.204: config changed from index 189 to 204, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New config: { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:13:01.866149 27154 raft_consensus.cc:2932] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Committing config change with OpId 2.204: config changed from index 189 to 204, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New config: { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:13:01.871289 26866 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eee65dca344f41528758a1babd90a2ff with cas_config_opid_index 189: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:01.875881 26873 catalog_manager.cc:5484] T eee65dca344f41528758a1babd90a2ff P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: config changed from index 189 to 204, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New cstate: current_term: 2 leader_uuid: "4165ee79ee03466981e067d998339611" committed_config { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "4165ee79ee03466981e067d998339611" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 41205 } attrs { promote: false } } }
I20240503 14:13:01.881242 27003 tablet_service.cc:1514] Processing DeleteTablet for tablet eee65dca344f41528758a1babd90a2ff with delete_type TABLET_DATA_TOMBSTONED (TS 820e18ddcc1e4d53821d43bc32652bc9 not found in new config with opid_index 204) from {username='slave'} at 127.0.0.1:40428
I20240503 14:13:01.882352 29680 tablet_replica.cc:330] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9: stopping tablet replica
I20240503 14:13:01.882645 29680 raft_consensus.cc:2229] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:01.882966 29680 raft_consensus.cc:2258] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:01.885079 29680 ts_tablet_manager.cc:1875] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:01.893159 29680 ts_tablet_manager.cc:1888] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.203
I20240503 14:13:01.893527 29680 log.cc:1196] T eee65dca344f41528758a1babd90a2ff P 820e18ddcc1e4d53821d43bc32652bc9: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/eee65dca344f41528758a1babd90a2ff
I20240503 14:13:01.894858 26866 catalog_manager.cc:4830] TS 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969): tablet eee65dca344f41528758a1babd90a2ff (table rebalance_test_table_0 [id=eda67e22fc6146289f3862c2470418fb]) successfully deleted
I20240503 14:13:02.872836 27132 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:02.959937 28321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:02.973007 28716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:03.009876 27003 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:03.042100 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:03.171557 28452 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:03.254398 27264 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:04.310674 27132 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:04.392407 28716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:13:04.399071 28276 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.70:44045, user_credentials={real_user=slave}} blocked reactor thread for 47060.5us
I20240503 14:13:04.412928 28321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:04.455351 27264 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:04.557523 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:04.559089 27003 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:04.609428 28452 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:06.575819 27132 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:06.608415 28321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:06.681668 28716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:06.704447 27003 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:06.839380 28452 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:06.862061 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:07.030046 27264 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:07.898300 27132 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:08.026732 28321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:08.126437 27003 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:08.149461 28716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:08.216759 28452 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:08.226562 27264 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:08.345541 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:08.972399 28321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:08.979017 27132 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:08.996059 27003 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:08.998467 28716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:09.027583 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:09.067242 27264 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:09.083266 28452 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
cf0ddf686eec42e5897a2ea24534acfb | 127.22.242.126:45965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.242.84:38033 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
305daa9a4d874d9db0e5378e5d27b617 | 127.22.242.66:38751 | HEALTHY | <none> | 3 | 0
4165ee79ee03466981e067d998339611 | 127.22.242.68:41205 | HEALTHY | <none> | 2 | 0
6b69a8c25e2b4de0bb10db8d7bdf83f0 | 127.22.242.71:42765 | HEALTHY | <none> | 0 | 0
820e18ddcc1e4d53821d43bc32652bc9 | 127.22.242.65:37969 | HEALTHY | <none> | 4 | 0
89a56aa776934c4582bfbdb0622b0cb3 | 127.22.242.70:44045 | HEALTHY | <none> | 1 | 0
cbb4bf53151a4061a81cf5bacf084516 | 127.22.242.69:39081 | HEALTHY | <none> | 1 | 0
f6c292faf79e49edabdbc7a310a30503 | 127.22.242.67:45423 | HEALTHY | <none> | 5 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.22.242.65 | experimental | 127.22.242.65:37969
local_ip_for_outbound_sockets | 127.22.242.66 | experimental | 127.22.242.66:38751
local_ip_for_outbound_sockets | 127.22.242.67 | experimental | 127.22.242.67:45423
local_ip_for_outbound_sockets | 127.22.242.68 | experimental | 127.22.242.68:41205
local_ip_for_outbound_sockets | 127.22.242.69 | experimental | 127.22.242.69:39081
local_ip_for_outbound_sockets | 127.22.242.70 | experimental | 127.22.242.70:44045
local_ip_for_outbound_sockets | 127.22.242.71 | experimental | 127.22.242.71:42765
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.242.65:37969
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.242.66:38751
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.242.67:45423
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.22.242.68:41205
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.242.69:39081
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.242.70:44045
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.22.242.71:42765
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.242.84:38033 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:13:09.124709 23497 log_verifier.cc:126] Checking tablet 09ca4d63eab743b3977a47bcf120b3cf
I20240503 14:13:09.140897 23497 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 09ca4d63eab743b3977a47bcf120b3cf
I20240503 14:13:09.141047 23497 log_verifier.cc:126] Checking tablet 0c41cab275df431f8ca3df66f9dcb871
I20240503 14:13:09.270290 23497 log_verifier.cc:177] Verified matching terms for 333 ops in tablet 0c41cab275df431f8ca3df66f9dcb871
I20240503 14:13:09.270774 23497 log_verifier.cc:126] Checking tablet 27288fd1dcc8421dab42d3c282f718ae
I20240503 14:13:09.388240 23497 log_verifier.cc:177] Verified matching terms for 305 ops in tablet 27288fd1dcc8421dab42d3c282f718ae
I20240503 14:13:09.388684 23497 log_verifier.cc:126] Checking tablet 41b3bc788e164ce99c05a492ee768724
I20240503 14:13:09.518424 23497 log_verifier.cc:177] Verified matching terms for 328 ops in tablet 41b3bc788e164ce99c05a492ee768724
I20240503 14:13:09.518880 23497 log_verifier.cc:126] Checking tablet 4cca566ca06643ffbc72f02be6d544b4
I20240503 14:13:09.686312 23497 log_verifier.cc:177] Verified matching terms for 422 ops in tablet 4cca566ca06643ffbc72f02be6d544b4
I20240503 14:13:09.686838 23497 log_verifier.cc:126] Checking tablet 619d33242fee4c73b1abc8eafb1c00e8
I20240503 14:13:09.835160 23497 log_verifier.cc:177] Verified matching terms for 339 ops in tablet 619d33242fee4c73b1abc8eafb1c00e8
I20240503 14:13:09.835599 23497 log_verifier.cc:126] Checking tablet 6385b8e7ad5849c9a17136f0967c516f
I20240503 14:13:09.975512 23497 log_verifier.cc:177] Verified matching terms for 336 ops in tablet 6385b8e7ad5849c9a17136f0967c516f
I20240503 14:13:09.976267 23497 log_verifier.cc:126] Checking tablet 7efea07ad3204b6e868a05e9d80f5b3a
I20240503 14:13:10.100153 23497 log_verifier.cc:177] Verified matching terms for 314 ops in tablet 7efea07ad3204b6e868a05e9d80f5b3a
I20240503 14:13:10.100579 23497 log_verifier.cc:126] Checking tablet 927839713c994a9d9ad9f071a6943b0d
I20240503 14:13:10.237514 23497 log_verifier.cc:177] Verified matching terms for 351 ops in tablet 927839713c994a9d9ad9f071a6943b0d
I20240503 14:13:10.249274 23497 log_verifier.cc:126] Checking tablet ab63a1d4c9764427a141feb6614fe18f
I20240503 14:13:10.359673 23497 log_verifier.cc:177] Verified matching terms for 313 ops in tablet ab63a1d4c9764427a141feb6614fe18f
I20240503 14:13:10.360059 23497 log_verifier.cc:126] Checking tablet c47ed0ca358f4370b61529f8ffba2503
I20240503 14:13:10.525087 23497 log_verifier.cc:177] Verified matching terms for 447 ops in tablet c47ed0ca358f4370b61529f8ffba2503
I20240503 14:13:10.525628 23497 log_verifier.cc:126] Checking tablet c6d0c7755fb8449ea077c3786dc619fc
I20240503 14:13:10.663017 23497 log_verifier.cc:177] Verified matching terms for 374 ops in tablet c6d0c7755fb8449ea077c3786dc619fc
I20240503 14:13:10.663486 23497 log_verifier.cc:126] Checking tablet eee65dca344f41528758a1babd90a2ff
I20240503 14:13:10.794670 23497 log_verifier.cc:177] Verified matching terms for 340 ops in tablet eee65dca344f41528758a1babd90a2ff
I20240503 14:13:10.795116 23497 log_verifier.cc:126] Checking tablet efa780a04e854838a933be5934cd1578
I20240503 14:13:10.936240 23497 log_verifier.cc:177] Verified matching terms for 350 ops in tablet efa780a04e854838a933be5934cd1578
I20240503 14:13:10.936669 23497 log_verifier.cc:126] Checking tablet f24cc60aba81429ca7275571faf77cb9
I20240503 14:13:11.090515 23497 log_verifier.cc:177] Verified matching terms for 418 ops in tablet f24cc60aba81429ca7275571faf77cb9
I20240503 14:13:11.091029 23497 log_verifier.cc:126] Checking tablet f2db6b6ac8bb4fffb18e0b65dea27b5c
I20240503 14:13:11.228617 23497 log_verifier.cc:177] Verified matching terms for 362 ops in tablet f2db6b6ac8bb4fffb18e0b65dea27b5c
I20240503 14:13:11.231935 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.72:0
--local_ip_for_outbound_sockets=127.22.242.72
--webserver_interface=127.22.242.72
--webserver_port=0
--tserver_master_addrs=127.22.242.126:45965
--builtin_ntp_servers=127.22.242.84:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:13:11.586555 29994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:11.586870 29994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:11.586957 29994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:11.594995 29994 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:11.595173 29994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:11.595422 29994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.72
I20240503 14:13:11.605252 29994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.242.72
--webserver_port=0
--tserver_master_addrs=127.22.242.126:45965
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.72
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:11.606676 29994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:11.608572 29994 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:11.621260 30003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:11.621212 30001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:11.621301 30000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:11.621788 29994 server_base.cc:1034] running on GCE node
I20240503 14:13:11.622452 29994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:11.623553 29994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:11.624859 29994 hybrid_clock.cc:648] HybridClock initialized: now 1714745591624791 us; error 56 us; skew 500 ppm
I20240503 14:13:11.625214 29994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:11.627591 29994 webserver.cc:460] Webserver started at http://127.22.242.72:46583/ using document root <none> and password file <none>
I20240503 14:13:11.628427 29994 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:11.628567 29994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:11.628919 29994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:11.631506 29994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "91ec783c4d7f433d877ee3724dc20b49"
format_stamp: "Formatted at 2024-05-03 14:13:11 on dist-test-slave-r3pd"
I20240503 14:13:11.632441 29994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "91ec783c4d7f433d877ee3724dc20b49"
format_stamp: "Formatted at 2024-05-03 14:13:11 on dist-test-slave-r3pd"
I20240503 14:13:11.638171 29994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 14:13:11.641673 30009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:11.643018 29994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:11.643194 29994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "91ec783c4d7f433d877ee3724dc20b49"
format_stamp: "Formatted at 2024-05-03 14:13:11 on dist-test-slave-r3pd"
I20240503 14:13:11.643393 29994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:11.665206 29994 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:11.666056 29994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:11.666316 29994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:11.667198 29994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:11.668628 29994 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:11.668730 29994 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:11.668843 29994 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:11.668922 29994 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:11.704372 29994 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.72:41237
I20240503 14:13:11.704424 30121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.72:41237 every 8 connection(s)
I20240503 14:13:11.705741 29994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:13:11.711575 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 29994
I20240503 14:13:11.711789 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 14:13:11.723969 30122 heartbeater.cc:347] Connected to a master server at 127.22.242.126:45965
I20240503 14:13:11.724357 30122 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:11.725416 30122 heartbeater.cc:511] Master 127.22.242.126:45965 requested a full tablet report, sending...
I20240503 14:13:11.727016 26882 ts_manager.cc:194] Registered new tserver with Master: 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72:41237)
I20240503 14:13:11.727890 26882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.72:36439
I20240503 14:13:12.325825 28321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:12.326052 27132 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:12.350931 28716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:12.372763 27003 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:12.402756 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:12.463523 27264 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:12.463551 28452 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:12.466149 30057 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:12.640863 27132 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:12.655958 28716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:12.665048 28452 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:12.667305 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:12.693876 28321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:12.695428 27003 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:12.696576 27264 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:12.702725 30057 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:12.731436 30122 heartbeater.cc:503] Master 127.22.242.126:45965 was elected leader, sending a full tablet report...
I20240503 14:13:12.820735 27021 consensus_queue.cc:237] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 339, Committed index: 339, Last appended: 1.339, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } }
I20240503 14:13:12.823559 27151 raft_consensus.cc:1261] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 339. Preceding OpId from leader: term: 1 index: 340. (index mismatch)
I20240503 14:13:12.824234 30217 consensus_queue.cc:1035] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 340, Last known committed idx: 339, Time since last communication: 0.000s
I20240503 14:13:12.826988 29553 raft_consensus.cc:2932] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Committing config change with OpId 1.340: config changed from index -1 to 340, NON_VOTER 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) added. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } } }
I20240503 14:13:12.827502 27151 raft_consensus.cc:2932] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Committing config change with OpId 1.340: config changed from index -1 to 340, NON_VOTER 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) added. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } } }
W20240503 14:13:12.832211 26956 consensus_peers.cc:485] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 -> Peer 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72:41237): Couldn't send request to peer 91ec783c4d7f433d877ee3724dc20b49. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 619d33242fee4c73b1abc8eafb1c00e8. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:12.832854 26874 catalog_manager.cc:5484] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: config changed from index -1 to 340, NON_VOTER 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) added. New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } } }
I20240503 14:13:12.850780 27151 consensus_queue.cc:237] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 2.351, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } }
I20240503 14:13:12.853310 28469 raft_consensus.cc:1261] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 2 FOLLOWER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 2 index: 351. Preceding OpId from leader: term: 2 index: 352. (index mismatch)
I20240503 14:13:12.854035 29880 consensus_queue.cc:1035] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20240503 14:13:12.857580 30231 raft_consensus.cc:2932] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 2 LEADER]: Committing config change with OpId 2.352: config changed from index 206 to 352, NON_VOTER 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) added. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } } }
I20240503 14:13:12.858224 28469 raft_consensus.cc:2932] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 2 FOLLOWER]: Committing config change with OpId 2.352: config changed from index 206 to 352, NON_VOTER 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) added. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } } }
I20240503 14:13:12.863399 26874 catalog_manager.cc:5484] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: config changed from index 206 to 352, NON_VOTER 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) added. New cstate: current_term: 2 leader_uuid: "305daa9a4d874d9db0e5378e5d27b617" committed_config { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:13:12.865880 27087 consensus_peers.cc:485] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 -> Peer 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72:41237): Couldn't send request to peer 91ec783c4d7f433d877ee3724dc20b49. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 927839713c994a9d9ad9f071a6943b0d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:12.877540 27282 consensus_queue.cc:237] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 1.350, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } }
I20240503 14:13:12.880005 28602 raft_consensus.cc:1261] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 1 index: 351. (index mismatch)
I20240503 14:13:12.880700 29849 consensus_queue.cc:1035] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20240503 14:13:12.894901 28602 raft_consensus.cc:2932] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.351: config changed from index 118 to 351, NON_VOTER 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) added. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } } }
I20240503 14:13:12.894649 30241 raft_consensus.cc:2932] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.351: config changed from index 118 to 351, NON_VOTER 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) added. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } } }
I20240503 14:13:12.900988 26882 catalog_manager.cc:5484] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 reported cstate change: config changed from index 118 to 351, NON_VOTER 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) added. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } } }
W20240503 14:13:12.906320 27217 consensus_peers.cc:485] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 -> Peer 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72:41237): Couldn't send request to peer 91ec783c4d7f433d877ee3724dc20b49. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efa780a04e854838a933be5934cd1578. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:12.915727 27021 consensus_queue.cc:237] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 314, Committed index: 314, Last appended: 1.314, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } }
I20240503 14:13:12.917933 28602 raft_consensus.cc:1261] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 314. Preceding OpId from leader: term: 1 index: 315. (index mismatch)
W20240503 14:13:12.918134 26956 consensus_peers.cc:485] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 -> Peer 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72:41237): Couldn't send request to peer 91ec783c4d7f433d877ee3724dc20b49. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7efea07ad3204b6e868a05e9d80f5b3a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:12.918561 29553 consensus_queue.cc:1035] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I20240503 14:13:12.921061 30220 raft_consensus.cc:2932] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Committing config change with OpId 1.315: config changed from index 102 to 315, NON_VOTER 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) added. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } } }
I20240503 14:13:12.921690 28602 raft_consensus.cc:2932] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.315: config changed from index 102 to 315, NON_VOTER 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) added. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } } }
I20240503 14:13:12.926844 26874 catalog_manager.cc:5484] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3 reported cstate change: config changed from index 102 to 315, NON_VOTER 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) added. New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } } }
I20240503 14:13:13.343962 30291 ts_tablet_manager.cc:902] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49: Initiating tablet copy from peer 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
I20240503 14:13:13.346350 30291 tablet_copy_client.cc:308] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:38751
I20240503 14:13:13.348628 30297 ts_tablet_manager.cc:902] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49: Initiating tablet copy from peer 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
I20240503 14:13:13.350383 30297 tablet_copy_client.cc:308] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:37969
I20240503 14:13:13.355314 27174 tablet_copy_service.cc:140] P 305daa9a4d874d9db0e5378e5d27b617: Received BeginTabletCopySession request for tablet 927839713c994a9d9ad9f071a6943b0d from peer 91ec783c4d7f433d877ee3724dc20b49 ({username='slave'} at 127.22.242.72:34353)
I20240503 14:13:13.355513 27174 tablet_copy_service.cc:161] P 305daa9a4d874d9db0e5378e5d27b617: Beginning new tablet copy session on tablet 927839713c994a9d9ad9f071a6943b0d from peer 91ec783c4d7f433d877ee3724dc20b49 at {username='slave'} at 127.22.242.72:34353: session id = 91ec783c4d7f433d877ee3724dc20b49-927839713c994a9d9ad9f071a6943b0d
I20240503 14:13:13.358223 27174 tablet_copy_source_session.cc:205] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:13.358788 27043 tablet_copy_service.cc:140] P 820e18ddcc1e4d53821d43bc32652bc9: Received BeginTabletCopySession request for tablet 619d33242fee4c73b1abc8eafb1c00e8 from peer 91ec783c4d7f433d877ee3724dc20b49 ({username='slave'} at 127.22.242.72:56559)
I20240503 14:13:13.358963 27043 tablet_copy_service.cc:161] P 820e18ddcc1e4d53821d43bc32652bc9: Beginning new tablet copy session on tablet 619d33242fee4c73b1abc8eafb1c00e8 from peer 91ec783c4d7f433d877ee3724dc20b49 at {username='slave'} at 127.22.242.72:56559: session id = 91ec783c4d7f433d877ee3724dc20b49-619d33242fee4c73b1abc8eafb1c00e8
I20240503 14:13:13.360956 27043 tablet_copy_source_session.cc:205] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:13.363018 30297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619d33242fee4c73b1abc8eafb1c00e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.363504 30291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927839713c994a9d9ad9f071a6943b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.374210 30303 ts_tablet_manager.cc:902] T 7efea07ad3204b6e868a05e9d80f5b3a P 91ec783c4d7f433d877ee3724dc20b49: Initiating tablet copy from peer 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969)
I20240503 14:13:13.376271 30303 tablet_copy_client.cc:308] T 7efea07ad3204b6e868a05e9d80f5b3a P 91ec783c4d7f433d877ee3724dc20b49: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:37969
I20240503 14:13:13.376943 30297 tablet_copy_client.cc:783] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:13.377189 30291 tablet_copy_client.cc:783] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:13.377187 27043 tablet_copy_service.cc:140] P 820e18ddcc1e4d53821d43bc32652bc9: Received BeginTabletCopySession request for tablet 7efea07ad3204b6e868a05e9d80f5b3a from peer 91ec783c4d7f433d877ee3724dc20b49 ({username='slave'} at 127.22.242.72:56559)
I20240503 14:13:13.377277 30297 tablet_copy_client.cc:647] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:13.377349 27043 tablet_copy_service.cc:161] P 820e18ddcc1e4d53821d43bc32652bc9: Beginning new tablet copy session on tablet 7efea07ad3204b6e868a05e9d80f5b3a from peer 91ec783c4d7f433d877ee3724dc20b49 at {username='slave'} at 127.22.242.72:56559: session id = 91ec783c4d7f433d877ee3724dc20b49-7efea07ad3204b6e868a05e9d80f5b3a
I20240503 14:13:13.377390 30291 tablet_copy_client.cc:647] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:13.379586 27043 tablet_copy_source_session.cc:205] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:13.381083 30297 tablet_copy_client.cc:523] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:13.381325 30303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7efea07ad3204b6e868a05e9d80f5b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.383311 30291 tablet_copy_client.cc:523] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:13.386372 30291 tablet_bootstrap.cc:492] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49: Bootstrap starting.
I20240503 14:13:13.386408 30297 tablet_bootstrap.cc:492] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49: Bootstrap starting.
I20240503 14:13:13.387996 30303 tablet_copy_client.cc:783] T 7efea07ad3204b6e868a05e9d80f5b3a P 91ec783c4d7f433d877ee3724dc20b49: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:13.388398 30303 tablet_copy_client.cc:647] T 7efea07ad3204b6e868a05e9d80f5b3a P 91ec783c4d7f433d877ee3724dc20b49: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:13.391399 30303 tablet_copy_client.cc:523] T 7efea07ad3204b6e868a05e9d80f5b3a P 91ec783c4d7f433d877ee3724dc20b49: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:13.394094 30303 tablet_bootstrap.cc:492] T 7efea07ad3204b6e868a05e9d80f5b3a P 91ec783c4d7f433d877ee3724dc20b49: Bootstrap starting.
I20240503 14:13:13.476085 30291 log.cc:824] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:13.488845 30311 ts_tablet_manager.cc:902] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49: Initiating tablet copy from peer f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423)
I20240503 14:13:13.531482 30311 tablet_copy_client.cc:308] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:45423
I20240503 14:13:13.551112 27304 tablet_copy_service.cc:140] P f6c292faf79e49edabdbc7a310a30503: Received BeginTabletCopySession request for tablet efa780a04e854838a933be5934cd1578 from peer 91ec783c4d7f433d877ee3724dc20b49 ({username='slave'} at 127.22.242.72:49333)
I20240503 14:13:13.551283 27304 tablet_copy_service.cc:161] P f6c292faf79e49edabdbc7a310a30503: Beginning new tablet copy session on tablet efa780a04e854838a933be5934cd1578 from peer 91ec783c4d7f433d877ee3724dc20b49 at {username='slave'} at 127.22.242.72:49333: session id = 91ec783c4d7f433d877ee3724dc20b49-efa780a04e854838a933be5934cd1578
I20240503 14:13:13.554190 27304 tablet_copy_source_session.cc:205] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:13.555678 30311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa780a04e854838a933be5934cd1578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.563087 30311 tablet_copy_client.cc:783] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:13.563409 30311 tablet_copy_client.cc:647] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:13.566695 30311 tablet_copy_client.cc:523] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:13.570441 30311 tablet_bootstrap.cc:492] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49: Bootstrap starting.
I20240503 14:13:13.938881 30303 tablet_bootstrap.cc:492] T 7efea07ad3204b6e868a05e9d80f5b3a P 91ec783c4d7f433d877ee3724dc20b49: Bootstrap replayed 1/1 log segments. Stats: ops{read=315 overwritten=0 applied=315 ignored=0} inserts{seen=943 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:13.939656 30303 tablet_bootstrap.cc:492] T 7efea07ad3204b6e868a05e9d80f5b3a P 91ec783c4d7f433d877ee3724dc20b49: Bootstrap complete.
I20240503 14:13:13.940331 30303 ts_tablet_manager.cc:1367] T 7efea07ad3204b6e868a05e9d80f5b3a P 91ec783c4d7f433d877ee3724dc20b49: Time spent bootstrapping tablet: real 0.546s user 0.349s sys 0.066s
I20240503 14:13:13.948985 30303 raft_consensus.cc:348] T 7efea07ad3204b6e868a05e9d80f5b3a P 91ec783c4d7f433d877ee3724dc20b49 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } }
I20240503 14:13:13.949616 30303 raft_consensus.cc:729] T 7efea07ad3204b6e868a05e9d80f5b3a P 91ec783c4d7f433d877ee3724dc20b49 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91ec783c4d7f433d877ee3724dc20b49, State: Initialized, Role: LEARNER
I20240503 14:13:13.950261 30303 consensus_queue.cc:260] T 7efea07ad3204b6e868a05e9d80f5b3a P 91ec783c4d7f433d877ee3724dc20b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 315, Last appended: 1.315, Last appended by leader: 315, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } }
I20240503 14:13:13.966991 30303 ts_tablet_manager.cc:1398] T 7efea07ad3204b6e868a05e9d80f5b3a P 91ec783c4d7f433d877ee3724dc20b49: Time spent starting tablet: real 0.026s user 0.005s sys 0.017s
I20240503 14:13:13.968319 27043 tablet_copy_service.cc:342] P 820e18ddcc1e4d53821d43bc32652bc9: Request end of tablet copy session 91ec783c4d7f433d877ee3724dc20b49-7efea07ad3204b6e868a05e9d80f5b3a received from {username='slave'} at 127.22.242.72:56559
I20240503 14:13:13.968452 27043 tablet_copy_service.cc:433] P 820e18ddcc1e4d53821d43bc32652bc9: ending tablet copy session 91ec783c4d7f433d877ee3724dc20b49-7efea07ad3204b6e868a05e9d80f5b3a on tablet 7efea07ad3204b6e868a05e9d80f5b3a with peer 91ec783c4d7f433d877ee3724dc20b49
I20240503 14:13:13.990959 30297 tablet_bootstrap.cc:492] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49: Bootstrap replayed 1/1 log segments. Stats: ops{read=340 overwritten=0 applied=340 ignored=0} inserts{seen=1012 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:13.991448 30297 tablet_bootstrap.cc:492] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49: Bootstrap complete.
I20240503 14:13:13.991780 30297 ts_tablet_manager.cc:1367] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49: Time spent bootstrapping tablet: real 0.606s user 0.388s sys 0.061s
I20240503 14:13:13.992812 30297 raft_consensus.cc:348] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } }
I20240503 14:13:13.993067 30297 raft_consensus.cc:729] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91ec783c4d7f433d877ee3724dc20b49, State: Initialized, Role: LEARNER
I20240503 14:13:13.993286 30297 consensus_queue.cc:260] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 340, Last appended: 1.340, Last appended by leader: 340, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } }
I20240503 14:13:13.994278 30297 ts_tablet_manager.cc:1398] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:13.995055 27043 tablet_copy_service.cc:342] P 820e18ddcc1e4d53821d43bc32652bc9: Request end of tablet copy session 91ec783c4d7f433d877ee3724dc20b49-619d33242fee4c73b1abc8eafb1c00e8 received from {username='slave'} at 127.22.242.72:56559
I20240503 14:13:13.995203 27043 tablet_copy_service.cc:433] P 820e18ddcc1e4d53821d43bc32652bc9: ending tablet copy session 91ec783c4d7f433d877ee3724dc20b49-619d33242fee4c73b1abc8eafb1c00e8 on tablet 619d33242fee4c73b1abc8eafb1c00e8 with peer 91ec783c4d7f433d877ee3724dc20b49
I20240503 14:13:14.072841 30291 tablet_bootstrap.cc:492] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49: Bootstrap replayed 1/1 log segments. Stats: ops{read=352 overwritten=0 applied=352 ignored=0} inserts{seen=1020 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:14.073524 30291 tablet_bootstrap.cc:492] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49: Bootstrap complete.
I20240503 14:13:14.073850 30291 ts_tablet_manager.cc:1367] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49: Time spent bootstrapping tablet: real 0.688s user 0.452s sys 0.049s
I20240503 14:13:14.074787 30291 raft_consensus.cc:348] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } }
I20240503 14:13:14.075078 30291 raft_consensus.cc:729] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 91ec783c4d7f433d877ee3724dc20b49, State: Initialized, Role: LEARNER
I20240503 14:13:14.075326 30291 consensus_queue.cc:260] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 352, Last appended: 2.352, Last appended by leader: 352, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } }
I20240503 14:13:14.075891 30291 ts_tablet_manager.cc:1398] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:14.076795 27174 tablet_copy_service.cc:342] P 305daa9a4d874d9db0e5378e5d27b617: Request end of tablet copy session 91ec783c4d7f433d877ee3724dc20b49-927839713c994a9d9ad9f071a6943b0d received from {username='slave'} at 127.22.242.72:34353
I20240503 14:13:14.077493 27174 tablet_copy_service.cc:433] P 305daa9a4d874d9db0e5378e5d27b617: ending tablet copy session 91ec783c4d7f433d877ee3724dc20b49-927839713c994a9d9ad9f071a6943b0d on tablet 927839713c994a9d9ad9f071a6943b0d with peer 91ec783c4d7f433d877ee3724dc20b49
I20240503 14:13:14.131887 30311 tablet_bootstrap.cc:492] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49: Bootstrap replayed 1/1 log segments. Stats: ops{read=351 overwritten=0 applied=351 ignored=0} inserts{seen=979 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:14.132328 30311 tablet_bootstrap.cc:492] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49: Bootstrap complete.
I20240503 14:13:14.132611 30311 ts_tablet_manager.cc:1367] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49: Time spent bootstrapping tablet: real 0.562s user 0.387s sys 0.058s
I20240503 14:13:14.133208 30311 raft_consensus.cc:348] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } }
I20240503 14:13:14.133416 30311 raft_consensus.cc:729] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91ec783c4d7f433d877ee3724dc20b49, State: Initialized, Role: LEARNER
I20240503 14:13:14.133606 30311 consensus_queue.cc:260] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 351, Last appended: 1.351, Last appended by leader: 351, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: true } }
I20240503 14:13:14.134083 30311 ts_tablet_manager.cc:1398] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:14.134932 27304 tablet_copy_service.cc:342] P f6c292faf79e49edabdbc7a310a30503: Request end of tablet copy session 91ec783c4d7f433d877ee3724dc20b49-efa780a04e854838a933be5934cd1578 received from {username='slave'} at 127.22.242.72:49333
I20240503 14:13:14.135099 27304 tablet_copy_service.cc:433] P f6c292faf79e49edabdbc7a310a30503: ending tablet copy session 91ec783c4d7f433d877ee3724dc20b49-efa780a04e854838a933be5934cd1578 on tablet efa780a04e854838a933be5934cd1578 with peer 91ec783c4d7f433d877ee3724dc20b49
I20240503 14:13:14.370640 30077 raft_consensus.cc:1203] T 7efea07ad3204b6e868a05e9d80f5b3a P 91ec783c4d7f433d877ee3724dc20b49 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.314->[1.315-1.315] Dedup: 1.315->[]
I20240503 14:13:14.401976 30077 raft_consensus.cc:1203] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.351->[2.352-2.352] Dedup: 2.352->[]
I20240503 14:13:14.464829 30077 raft_consensus.cc:1203] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.339->[1.340-1.340] Dedup: 1.340->[]
I20240503 14:13:14.479899 30077 raft_consensus.cc:1203] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.350->[1.351-1.351] Dedup: 1.351->[]
I20240503 14:13:14.872391 30352 raft_consensus.cc:1050] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617: attempting to promote NON_VOTER 91ec783c4d7f433d877ee3724dc20b49 to VOTER
I20240503 14:13:14.873160 30352 consensus_queue.cc:237] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 352, Committed index: 352, Last appended: 2.352, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } }
I20240503 14:13:14.875332 30077 raft_consensus.cc:1261] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49 [term 2 LEARNER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 2 index: 352. Preceding OpId from leader: term: 2 index: 353. (index mismatch)
I20240503 14:13:14.875337 28469 raft_consensus.cc:1261] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 2 FOLLOWER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 2 index: 352. Preceding OpId from leader: term: 2 index: 353. (index mismatch)
I20240503 14:13:14.875834 30231 consensus_queue.cc:1035] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 352, Time since last communication: 0.000s
I20240503 14:13:14.876145 30352 consensus_queue.cc:1035] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 352, Time since last communication: 0.000s
I20240503 14:13:14.878940 30231 raft_consensus.cc:2932] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 2 LEADER]: Committing config change with OpId 2.353: config changed from index 352 to 353, 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:14.879670 28469 raft_consensus.cc:2932] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 2 FOLLOWER]: Committing config change with OpId 2.353: config changed from index 352 to 353, 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:14.880721 30077 raft_consensus.cc:2932] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49 [term 2 FOLLOWER]: Committing config change with OpId 2.353: config changed from index 352 to 353, 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:14.884189 26882 catalog_manager.cc:5484] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: config changed from index 352 to 353, 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "305daa9a4d874d9db0e5378e5d27b617" committed_config { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:14.901202 30353 raft_consensus.cc:1050] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9: attempting to promote NON_VOTER 91ec783c4d7f433d877ee3724dc20b49 to VOTER
I20240503 14:13:14.902043 30353 consensus_queue.cc:237] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 1.315, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } }
I20240503 14:13:14.904001 28602 raft_consensus.cc:1261] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 315. Preceding OpId from leader: term: 1 index: 316. (index mismatch)
I20240503 14:13:14.904217 30077 raft_consensus.cc:1261] T 7efea07ad3204b6e868a05e9d80f5b3a P 91ec783c4d7f433d877ee3724dc20b49 [term 1 LEARNER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 315. Preceding OpId from leader: term: 1 index: 316. (index mismatch)
I20240503 14:13:14.904582 29553 consensus_queue.cc:1035] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20240503 14:13:14.904939 30353 consensus_queue.cc:1035] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20240503 14:13:14.907613 30351 raft_consensus.cc:2932] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Committing config change with OpId 1.316: config changed from index 315 to 316, 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:14.908254 30077 raft_consensus.cc:2932] T 7efea07ad3204b6e868a05e9d80f5b3a P 91ec783c4d7f433d877ee3724dc20b49 [term 1 FOLLOWER]: Committing config change with OpId 1.316: config changed from index 315 to 316, 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:14.911223 28602 raft_consensus.cc:2932] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.316: config changed from index 315 to 316, 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:14.912757 26874 catalog_manager.cc:5484] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 reported cstate change: config changed from index 315 to 316, 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:14.917096 27021 consensus_queue.cc:237] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 316, Committed index: 316, Last appended: 1.316, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } }
I20240503 14:13:14.918373 29553 raft_consensus.cc:1050] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9: attempting to promote NON_VOTER 91ec783c4d7f433d877ee3724dc20b49 to VOTER
I20240503 14:13:14.919124 29553 consensus_queue.cc:237] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 1.340, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } }
I20240503 14:13:14.919611 30077 raft_consensus.cc:1261] T 7efea07ad3204b6e868a05e9d80f5b3a P 91ec783c4d7f433d877ee3724dc20b49 [term 1 FOLLOWER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 316. Preceding OpId from leader: term: 1 index: 317. (index mismatch)
I20240503 14:13:14.920360 30443 consensus_queue.cc:1035] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I20240503 14:13:14.921976 27151 raft_consensus.cc:1261] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 340. Preceding OpId from leader: term: 1 index: 341. (index mismatch)
I20240503 14:13:14.922433 30077 raft_consensus.cc:1261] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49 [term 1 LEARNER]: Refusing update from remote peer 820e18ddcc1e4d53821d43bc32652bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 340. Preceding OpId from leader: term: 1 index: 341. (index mismatch)
I20240503 14:13:14.922926 30443 consensus_queue.cc:1035] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 341, Last known committed idx: 340, Time since last communication: 0.000s
I20240503 14:13:14.923287 30351 consensus_queue.cc:1035] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 341, Last known committed idx: 340, Time since last communication: 0.000s
I20240503 14:13:14.923377 29553 raft_consensus.cc:2932] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Committing config change with OpId 1.317: config changed from index 316 to 317, VOTER 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) evicted. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:14.924067 30076 raft_consensus.cc:2932] T 7efea07ad3204b6e868a05e9d80f5b3a P 91ec783c4d7f433d877ee3724dc20b49 [term 1 FOLLOWER]: Committing config change with OpId 1.317: config changed from index 316 to 317, VOTER 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) evicted. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:14.926060 30351 raft_consensus.cc:2932] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Committing config change with OpId 1.341: config changed from index 340 to 341, 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:14.927680 26866 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7efea07ad3204b6e868a05e9d80f5b3a with cas_config_opid_index 316: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:14.929141 26882 catalog_manager.cc:5484] T 7efea07ad3204b6e868a05e9d80f5b3a P 820e18ddcc1e4d53821d43bc32652bc9 reported cstate change: config changed from index 316 to 317, VOTER 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70) evicted. New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:14.929764 30077 raft_consensus.cc:2932] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49 [term 1 FOLLOWER]: Committing config change with OpId 1.341: config changed from index 340 to 341, 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:14.931521 27151 raft_consensus.cc:2932] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Committing config change with OpId 1.341: config changed from index 340 to 341, 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:14.936730 28584 tablet_service.cc:1514] Processing DeleteTablet for tablet 7efea07ad3204b6e868a05e9d80f5b3a with delete_type TABLET_DATA_TOMBSTONED (TS 89a56aa776934c4582bfbdb0622b0cb3 not found in new config with opid_index 317) from {username='slave'} at 127.0.0.1:57606
I20240503 14:13:14.936647 26874 catalog_manager.cc:5484] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: config changed from index 340 to 341, 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "820e18ddcc1e4d53821d43bc32652bc9" committed_config { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:14.938057 30447 tablet_replica.cc:330] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3: stopping tablet replica
I20240503 14:13:14.938546 30447 raft_consensus.cc:2229] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:14.938959 30447 raft_consensus.cc:2258] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:14.941540 30447 ts_tablet_manager.cc:1875] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:14.946005 30447 ts_tablet_manager.cc:1888] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.316
I20240503 14:13:14.946167 30447 log.cc:1196] T 7efea07ad3204b6e868a05e9d80f5b3a P 89a56aa776934c4582bfbdb0622b0cb3: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal/wals/7efea07ad3204b6e868a05e9d80f5b3a
I20240503 14:13:14.946933 26868 catalog_manager.cc:4830] TS 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70:44045): tablet 7efea07ad3204b6e868a05e9d80f5b3a (table rebalance_test_table_0 [id=eda67e22fc6146289f3862c2470418fb]) successfully deleted
I20240503 14:13:15.031922 30464 raft_consensus.cc:1050] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503: attempting to promote NON_VOTER 91ec783c4d7f433d877ee3724dc20b49 to VOTER
I20240503 14:13:15.032627 30464 consensus_queue.cc:237] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 1.351, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } }
I20240503 14:13:15.034807 28602 raft_consensus.cc:1261] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 351. Preceding OpId from leader: term: 1 index: 352. (index mismatch)
I20240503 14:13:15.034783 30077 raft_consensus.cc:1261] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [term 1 LEARNER]: Refusing update from remote peer f6c292faf79e49edabdbc7a310a30503: Log matching property violated. Preceding OpId in replica: term: 1 index: 351. Preceding OpId from leader: term: 1 index: 352. (index mismatch)
I20240503 14:13:15.035377 30463 consensus_queue.cc:1035] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20240503 14:13:15.036012 30362 consensus_queue.cc:1035] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20240503 14:13:15.039170 30463 raft_consensus.cc:2932] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Committing config change with OpId 1.352: config changed from index 351 to 352, 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:15.040486 27021 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "619d33242fee4c73b1abc8eafb1c00e8"
dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39106
I20240503 14:13:15.040699 27021 raft_consensus.cc:595] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:15.041028 28602 raft_consensus.cc:2932] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.352: config changed from index 351 to 352, 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:15.055909 26882 catalog_manager.cc:5484] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 reported cstate change: config changed from index 351 to 352, 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f6c292faf79e49edabdbc7a310a30503" committed_config { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:15.055821 30076 raft_consensus.cc:2932] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [term 1 FOLLOWER]: Committing config change with OpId 1.352: config changed from index 351 to 352, 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:15.064270 27151 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "927839713c994a9d9ad9f071a6943b0d"
dest_uuid: "305daa9a4d874d9db0e5378e5d27b617"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57884
I20240503 14:13:15.064478 27151 raft_consensus.cc:595] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 2 LEADER]: Received request to transfer leadership
I20240503 14:13:15.082345 27282 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "efa780a04e854838a933be5934cd1578"
dest_uuid: "f6c292faf79e49edabdbc7a310a30503"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42946
I20240503 14:13:15.082556 27282 raft_consensus.cc:595] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:15.100725 27021 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "619d33242fee4c73b1abc8eafb1c00e8"
dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39134
I20240503 14:13:15.100921 27021 raft_consensus.cc:595] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:15.119520 27151 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "927839713c994a9d9ad9f071a6943b0d"
dest_uuid: "305daa9a4d874d9db0e5378e5d27b617"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57912
I20240503 14:13:15.119757 27151 raft_consensus.cc:595] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 2 LEADER]: Received request to transfer leadership
I20240503 14:13:15.260862 30231 raft_consensus.cc:979] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617: : Instructing follower cbb4bf53151a4061a81cf5bacf084516 to start an election
I20240503 14:13:15.261174 30435 raft_consensus.cc:1067] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 2 LEADER]: Signalling peer cbb4bf53151a4061a81cf5bacf084516 to start an election
I20240503 14:13:15.261839 28469 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "927839713c994a9d9ad9f071a6943b0d"
dest_uuid: "cbb4bf53151a4061a81cf5bacf084516"
from {username='slave'} at 127.22.242.66:42507
I20240503 14:13:15.262048 28469 raft_consensus.cc:482] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:15.262194 28469 raft_consensus.cc:3037] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:13:15.264063 28469 raft_consensus.cc:504] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } }
I20240503 14:13:15.264801 28469 leader_election.cc:288] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [CANDIDATE]: Term 3 election: Requested vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751), 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72:41237)
I20240503 14:13:15.265858 27151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927839713c994a9d9ad9f071a6943b0d" candidate_uuid: "cbb4bf53151a4061a81cf5bacf084516" candidate_term: 3 candidate_status { last_received { term: 2 index: 353 } } ignore_live_leader: true dest_uuid: "305daa9a4d874d9db0e5378e5d27b617"
I20240503 14:13:15.266106 27151 raft_consensus.cc:3032] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:13:15.266207 27151 raft_consensus.cc:729] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 305daa9a4d874d9db0e5378e5d27b617, State: Running, Role: LEADER
I20240503 14:13:15.266626 27151 consensus_queue.cc:260] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 2.353, Last appended by leader: 353, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } }
I20240503 14:13:15.266952 27151 raft_consensus.cc:3037] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:13:15.268868 27151 raft_consensus.cc:2454] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbb4bf53151a4061a81cf5bacf084516 in term 3.
I20240503 14:13:15.269258 28407 leader_election.cc:302] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, cbb4bf53151a4061a81cf5bacf084516; no voters:
I20240503 14:13:15.269546 30411 raft_consensus.cc:2781] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:13:15.269954 30411 raft_consensus.cc:686] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 3 LEADER]: Becoming Leader. State: Replica: cbb4bf53151a4061a81cf5bacf084516, State: Running, Role: LEADER
I20240503 14:13:15.270346 30411 consensus_queue.cc:237] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 2.353, Last appended by leader: 353, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } }
I20240503 14:13:15.272236 30076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927839713c994a9d9ad9f071a6943b0d" candidate_uuid: "cbb4bf53151a4061a81cf5bacf084516" candidate_term: 3 candidate_status { last_received { term: 2 index: 353 } } ignore_live_leader: true dest_uuid: "91ec783c4d7f433d877ee3724dc20b49"
I20240503 14:13:15.272505 30076 raft_consensus.cc:3037] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:13:15.272856 26882 catalog_manager.cc:5484] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 reported cstate change: term changed from 2 to 3, leader changed from 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) to cbb4bf53151a4061a81cf5bacf084516 (127.22.242.69). New cstate: current_term: 3 leader_uuid: "cbb4bf53151a4061a81cf5bacf084516" committed_config { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:15.274523 30076 raft_consensus.cc:2454] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbb4bf53151a4061a81cf5bacf084516 in term 3.
I20240503 14:13:15.338197 27282 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "efa780a04e854838a933be5934cd1578"
dest_uuid: "f6c292faf79e49edabdbc7a310a30503"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42970
I20240503 14:13:15.338405 27282 raft_consensus.cc:595] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:15.351188 29553 raft_consensus.cc:979] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9: : Instructing follower 305daa9a4d874d9db0e5378e5d27b617 to start an election
I20240503 14:13:15.351389 29553 raft_consensus.cc:1067] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Signalling peer 305daa9a4d874d9db0e5378e5d27b617 to start an election
I20240503 14:13:15.352021 27151 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "619d33242fee4c73b1abc8eafb1c00e8"
dest_uuid: "305daa9a4d874d9db0e5378e5d27b617"
from {username='slave'} at 127.22.242.65:54731
I20240503 14:13:15.352285 27151 raft_consensus.cc:482] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:15.352398 27151 raft_consensus.cc:3037] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:15.354969 27151 raft_consensus.cc:504] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } }
I20240503 14:13:15.356009 27021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619d33242fee4c73b1abc8eafb1c00e8" candidate_uuid: "305daa9a4d874d9db0e5378e5d27b617" candidate_term: 2 candidate_status { last_received { term: 1 index: 341 } } ignore_live_leader: true dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9"
I20240503 14:13:15.356271 27021 raft_consensus.cc:3032] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:15.356390 27021 raft_consensus.cc:729] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 820e18ddcc1e4d53821d43bc32652bc9, State: Running, Role: LEADER
I20240503 14:13:15.356700 27021 consensus_queue.cc:260] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 341, Committed index: 341, Last appended: 1.341, Last appended by leader: 341, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } }
I20240503 14:13:15.357026 27021 raft_consensus.cc:3037] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:15.359061 27021 raft_consensus.cc:2454] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 305daa9a4d874d9db0e5378e5d27b617 in term 2.
I20240503 14:13:15.359472 27087 leader_election.cc:302] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 305daa9a4d874d9db0e5378e5d27b617, 820e18ddcc1e4d53821d43bc32652bc9; no voters:
I20240503 14:13:15.359628 27021 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "619d33242fee4c73b1abc8eafb1c00e8"
dest_uuid: "820e18ddcc1e4d53821d43bc32652bc9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39160
I20240503 14:13:15.359681 27151 leader_election.cc:288] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [CANDIDATE]: Term 2 election: Requested vote from peers 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969), 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72:41237)
I20240503 14:13:15.359794 27021 raft_consensus.cc:595] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Received request to transfer leadership
I20240503 14:13:15.359839 30435 raft_consensus.cc:2781] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:15.359898 27021 raft_consensus.cc:603] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240503 14:13:15.360003 30435 raft_consensus.cc:686] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [term 2 LEADER]: Becoming Leader. State: Replica: 305daa9a4d874d9db0e5378e5d27b617, State: Running, Role: LEADER
I20240503 14:13:15.360339 30076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619d33242fee4c73b1abc8eafb1c00e8" candidate_uuid: "305daa9a4d874d9db0e5378e5d27b617" candidate_term: 2 candidate_status { last_received { term: 1 index: 341 } } ignore_live_leader: true dest_uuid: "91ec783c4d7f433d877ee3724dc20b49"
I20240503 14:13:15.360527 30076 raft_consensus.cc:3037] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:15.360339 30435 consensus_queue.cc:237] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 341, Committed index: 341, Last appended: 1.341, Last appended by leader: 341, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } }
I20240503 14:13:15.363255 30076 raft_consensus.cc:2454] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 305daa9a4d874d9db0e5378e5d27b617 in term 2.
I20240503 14:13:15.364758 26882 catalog_manager.cc:5484] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: term changed from 1 to 2, leader changed from 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) to 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66). New cstate: current_term: 2 leader_uuid: "305daa9a4d874d9db0e5378e5d27b617" committed_config { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:15.590874 27282 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "efa780a04e854838a933be5934cd1578"
dest_uuid: "f6c292faf79e49edabdbc7a310a30503"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42994
I20240503 14:13:15.591137 27282 raft_consensus.cc:595] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:15.594736 30553 raft_consensus.cc:979] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503: : Instructing follower 91ec783c4d7f433d877ee3724dc20b49 to start an election
I20240503 14:13:15.594933 30362 raft_consensus.cc:1067] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Signalling peer 91ec783c4d7f433d877ee3724dc20b49 to start an election
I20240503 14:13:15.595808 30076 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "efa780a04e854838a933be5934cd1578"
dest_uuid: "91ec783c4d7f433d877ee3724dc20b49"
from {username='slave'} at 127.22.242.67:57491
I20240503 14:13:15.596181 30076 raft_consensus.cc:482] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:15.596352 30076 raft_consensus.cc:3037] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:15.598637 30076 raft_consensus.cc:504] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } }
I20240503 14:13:15.600052 30076 leader_election.cc:288] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [CANDIDATE]: Term 2 election: Requested vote from peers f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423), 89a56aa776934c4582bfbdb0622b0cb3 (127.22.242.70:44045)
I20240503 14:13:15.600644 27282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa780a04e854838a933be5934cd1578" candidate_uuid: "91ec783c4d7f433d877ee3724dc20b49" candidate_term: 2 candidate_status { last_received { term: 1 index: 352 } } ignore_live_leader: true dest_uuid: "f6c292faf79e49edabdbc7a310a30503"
I20240503 14:13:15.600992 27282 raft_consensus.cc:3032] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:15.601083 27282 raft_consensus.cc:729] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6c292faf79e49edabdbc7a310a30503, State: Running, Role: LEADER
I20240503 14:13:15.601392 27282 consensus_queue.cc:260] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 352, Committed index: 352, Last appended: 1.352, Last appended by leader: 352, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } }
I20240503 14:13:15.601795 27282 raft_consensus.cc:3037] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:15.605480 27282 raft_consensus.cc:2454] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91ec783c4d7f433d877ee3724dc20b49 in term 2.
I20240503 14:13:15.606070 30010 leader_election.cc:302] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91ec783c4d7f433d877ee3724dc20b49, f6c292faf79e49edabdbc7a310a30503; no voters:
I20240503 14:13:15.607115 30561 raft_consensus.cc:2781] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:15.608019 30561 raft_consensus.cc:686] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [term 2 LEADER]: Becoming Leader. State: Replica: 91ec783c4d7f433d877ee3724dc20b49, State: Running, Role: LEADER
I20240503 14:13:15.608801 30561 consensus_queue.cc:237] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 352, Committed index: 352, Last appended: 1.352, Last appended by leader: 352, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } }
I20240503 14:13:15.611802 28602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa780a04e854838a933be5934cd1578" candidate_uuid: "91ec783c4d7f433d877ee3724dc20b49" candidate_term: 2 candidate_status { last_received { term: 1 index: 352 } } ignore_live_leader: true dest_uuid: "89a56aa776934c4582bfbdb0622b0cb3"
I20240503 14:13:15.612082 28602 raft_consensus.cc:3037] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:15.614009 26882 catalog_manager.cc:5484] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 reported cstate change: term changed from 1 to 2, leader changed from f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) to 91ec783c4d7f433d877ee3724dc20b49 (127.22.242.72). New cstate: current_term: 2 leader_uuid: "91ec783c4d7f433d877ee3724dc20b49" committed_config { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:15.614627 28602 raft_consensus.cc:2454] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91ec783c4d7f433d877ee3724dc20b49 in term 2.
I20240503 14:13:15.650808 27151 raft_consensus.cc:1261] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 3 FOLLOWER]: Refusing update from remote peer cbb4bf53151a4061a81cf5bacf084516: Log matching property violated. Preceding OpId in replica: term: 2 index: 353. Preceding OpId from leader: term: 3 index: 354. (index mismatch)
I20240503 14:13:15.651440 30411 consensus_queue.cc:1035] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20240503 14:13:15.655877 30076 raft_consensus.cc:1261] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49 [term 3 FOLLOWER]: Refusing update from remote peer cbb4bf53151a4061a81cf5bacf084516: Log matching property violated. Preceding OpId in replica: term: 2 index: 353. Preceding OpId from leader: term: 3 index: 354. (index mismatch)
I20240503 14:13:15.656935 30571 consensus_queue.cc:1035] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20240503 14:13:15.815392 30076 raft_consensus.cc:1261] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49 [term 2 FOLLOWER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 1 index: 341. Preceding OpId from leader: term: 2 index: 342. (index mismatch)
I20240503 14:13:15.816030 30435 consensus_queue.cc:1035] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 342, Last known committed idx: 341, Time since last communication: 0.000s
I20240503 14:13:15.819413 27021 raft_consensus.cc:1261] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 1 index: 341. Preceding OpId from leader: term: 2 index: 342. (index mismatch)
I20240503 14:13:15.820250 30231 consensus_queue.cc:1035] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e18ddcc1e4d53821d43bc32652bc9" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 37969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 342, Last known committed idx: 341, Time since last communication: 0.000s
I20240503 14:13:15.823454 27151 consensus_queue.cc:237] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 342, Committed index: 342, Last appended: 2.342, Last appended by leader: 341, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } }
I20240503 14:13:15.825452 30076 raft_consensus.cc:1261] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49 [term 2 FOLLOWER]: Refusing update from remote peer 305daa9a4d874d9db0e5378e5d27b617: Log matching property violated. Preceding OpId in replica: term: 2 index: 342. Preceding OpId from leader: term: 2 index: 343. (index mismatch)
I20240503 14:13:15.826159 30573 consensus_queue.cc:1035] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 343, Last known committed idx: 342, Time since last communication: 0.000s
I20240503 14:13:15.828665 30231 raft_consensus.cc:2932] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 [term 2 LEADER]: Committing config change with OpId 2.343: config changed from index 341 to 343, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:15.829283 30076 raft_consensus.cc:2932] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49 [term 2 FOLLOWER]: Committing config change with OpId 2.343: config changed from index 341 to 343, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:15.832753 26868 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 619d33242fee4c73b1abc8eafb1c00e8 with cas_config_opid_index 341: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:15.834509 26876 catalog_manager.cc:5484] T 619d33242fee4c73b1abc8eafb1c00e8 P 305daa9a4d874d9db0e5378e5d27b617 reported cstate change: config changed from index 341 to 343, VOTER 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65) evicted. New cstate: current_term: 2 leader_uuid: "305daa9a4d874d9db0e5378e5d27b617" committed_config { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:15.838295 27003 tablet_service.cc:1514] Processing DeleteTablet for tablet 619d33242fee4c73b1abc8eafb1c00e8 with delete_type TABLET_DATA_TOMBSTONED (TS 820e18ddcc1e4d53821d43bc32652bc9 not found in new config with opid_index 343) from {username='slave'} at 127.0.0.1:40428
I20240503 14:13:15.838944 30581 tablet_replica.cc:330] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9: stopping tablet replica
I20240503 14:13:15.839238 30581 raft_consensus.cc:2229] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:15.839460 30581 raft_consensus.cc:2258] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:15.841074 30581 ts_tablet_manager.cc:1875] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:15.846359 30581 ts_tablet_manager.cc:1888] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.342
I20240503 14:13:15.846472 30581 log.cc:1196] T 619d33242fee4c73b1abc8eafb1c00e8 P 820e18ddcc1e4d53821d43bc32652bc9: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/619d33242fee4c73b1abc8eafb1c00e8
I20240503 14:13:15.847867 26866 catalog_manager.cc:4830] TS 820e18ddcc1e4d53821d43bc32652bc9 (127.22.242.65:37969): tablet 619d33242fee4c73b1abc8eafb1c00e8 (table rebalance_test_table_4 [id=6cdd58518c114d93a32a80b11512af8f]) successfully deleted
I20240503 14:13:16.008441 28602 raft_consensus.cc:1261] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 [term 2 FOLLOWER]: Refusing update from remote peer 91ec783c4d7f433d877ee3724dc20b49: Log matching property violated. Preceding OpId in replica: term: 1 index: 352. Preceding OpId from leader: term: 2 index: 353. (index mismatch)
I20240503 14:13:16.009172 30561 consensus_queue.cc:1035] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 352, Time since last communication: 0.000s
I20240503 14:13:16.013792 27282 raft_consensus.cc:1261] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 2 FOLLOWER]: Refusing update from remote peer 91ec783c4d7f433d877ee3724dc20b49: Log matching property violated. Preceding OpId in replica: term: 1 index: 352. Preceding OpId from leader: term: 2 index: 353. (index mismatch)
I20240503 14:13:16.014437 30561 consensus_queue.cc:1035] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c292faf79e49edabdbc7a310a30503" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 45423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 352, Time since last communication: 0.000s
I20240503 14:13:16.023696 30076 consensus_queue.cc:237] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 2.353, Last appended by leader: 352, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } }
I20240503 14:13:16.025298 28602 raft_consensus.cc:1261] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 [term 2 FOLLOWER]: Refusing update from remote peer 91ec783c4d7f433d877ee3724dc20b49: Log matching property violated. Preceding OpId in replica: term: 2 index: 353. Preceding OpId from leader: term: 2 index: 354. (index mismatch)
I20240503 14:13:16.025816 30562 consensus_queue.cc:1035] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20240503 14:13:16.027832 30561 raft_consensus.cc:2932] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 [term 2 LEADER]: Committing config change with OpId 2.354: config changed from index 352 to 354, VOTER f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) evicted. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:16.028681 28602 raft_consensus.cc:2932] T efa780a04e854838a933be5934cd1578 P 89a56aa776934c4582bfbdb0622b0cb3 [term 2 FOLLOWER]: Committing config change with OpId 2.354: config changed from index 352 to 354, VOTER f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) evicted. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:16.031718 26866 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet efa780a04e854838a933be5934cd1578 with cas_config_opid_index 352: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:16.033519 26876 catalog_manager.cc:5484] T efa780a04e854838a933be5934cd1578 P 91ec783c4d7f433d877ee3724dc20b49 reported cstate change: config changed from index 352 to 354, VOTER f6c292faf79e49edabdbc7a310a30503 (127.22.242.67) evicted. New cstate: current_term: 2 leader_uuid: "91ec783c4d7f433d877ee3724dc20b49" committed_config { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "89a56aa776934c4582bfbdb0622b0cb3" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 44045 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:16.036963 27264 tablet_service.cc:1514] Processing DeleteTablet for tablet efa780a04e854838a933be5934cd1578 with delete_type TABLET_DATA_TOMBSTONED (TS f6c292faf79e49edabdbc7a310a30503 not found in new config with opid_index 354) from {username='slave'} at 127.0.0.1:41116
I20240503 14:13:16.037698 30606 tablet_replica.cc:330] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503: stopping tablet replica
I20240503 14:13:16.038038 30606 raft_consensus.cc:2229] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:16.038235 30606 raft_consensus.cc:2258] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:16.039917 30606 ts_tablet_manager.cc:1875] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:16.044384 30606 ts_tablet_manager.cc:1888] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.353
I20240503 14:13:16.044508 30606 log.cc:1196] T efa780a04e854838a933be5934cd1578 P f6c292faf79e49edabdbc7a310a30503: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/efa780a04e854838a933be5934cd1578
I20240503 14:13:16.045300 26866 catalog_manager.cc:4830] TS f6c292faf79e49edabdbc7a310a30503 (127.22.242.67:45423): tablet efa780a04e854838a933be5934cd1578 (table rebalance_test_table_1 [id=34a5ee8103e54b178f47b78610c6a027]) successfully deleted
I20240503 14:13:16.112535 28469 consensus_queue.cc:237] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 354, Committed index: 354, Last appended: 3.354, Last appended by leader: 353, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } }
I20240503 14:13:16.114002 30076 raft_consensus.cc:1261] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49 [term 3 FOLLOWER]: Refusing update from remote peer cbb4bf53151a4061a81cf5bacf084516: Log matching property violated. Preceding OpId in replica: term: 3 index: 354. Preceding OpId from leader: term: 3 index: 355. (index mismatch)
I20240503 14:13:16.114527 30516 consensus_queue.cc:1035] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 355, Last known committed idx: 354, Time since last communication: 0.000s
I20240503 14:13:16.116703 30571 raft_consensus.cc:2932] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 [term 3 LEADER]: Committing config change with OpId 3.355: config changed from index 353 to 355, VOTER 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) evicted. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:16.117264 30076 raft_consensus.cc:2932] T 927839713c994a9d9ad9f071a6943b0d P 91ec783c4d7f433d877ee3724dc20b49 [term 3 FOLLOWER]: Committing config change with OpId 3.355: config changed from index 353 to 355, VOTER 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) evicted. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } } }
I20240503 14:13:16.119840 26868 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 927839713c994a9d9ad9f071a6943b0d with cas_config_opid_index 353: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:16.120990 26876 catalog_manager.cc:5484] T 927839713c994a9d9ad9f071a6943b0d P cbb4bf53151a4061a81cf5bacf084516 reported cstate change: config changed from index 353 to 355, VOTER 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66) evicted. New cstate: current_term: 3 leader_uuid: "cbb4bf53151a4061a81cf5bacf084516" committed_config { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "cbb4bf53151a4061a81cf5bacf084516" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 39081 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:16.124279 27132 tablet_service.cc:1514] Processing DeleteTablet for tablet 927839713c994a9d9ad9f071a6943b0d with delete_type TABLET_DATA_TOMBSTONED (TS 305daa9a4d874d9db0e5378e5d27b617 not found in new config with opid_index 355) from {username='slave'} at 127.0.0.1:54420
I20240503 14:13:16.125010 30622 tablet_replica.cc:330] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617: stopping tablet replica
I20240503 14:13:16.125205 30622 raft_consensus.cc:2229] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:16.125444 30622 raft_consensus.cc:2258] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:16.127146 30622 ts_tablet_manager.cc:1875] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:16.131793 30622 ts_tablet_manager.cc:1888] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.354
I20240503 14:13:16.131906 30622 log.cc:1196] T 927839713c994a9d9ad9f071a6943b0d P 305daa9a4d874d9db0e5378e5d27b617: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/927839713c994a9d9ad9f071a6943b0d
I20240503 14:13:16.132709 26868 catalog_manager.cc:4830] TS 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751): tablet 927839713c994a9d9ad9f071a6943b0d (table rebalance_test_table_2 [id=8c704c1747ee4a41b347496dc856e7bf]) successfully deleted
I20240503 14:13:16.431582 27132 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:16.449051 28716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:16.481156 28321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:16.525239 30057 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:16.545003 28452 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:16.554332 27003 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:16.555497 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:16.582979 27264 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:16.741014 27132 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:16.744596 28716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:16.755086 28321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:16.785965 27003 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:16.790714 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:16.819201 30057 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:16.865059 28452 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:16.865059 27264 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:17.164475 27132 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:17.164666 28321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:17.168637 28716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:17.183367 27003 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:17.190025 30057 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:17.191833 28452 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:17.203200 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:17.204942 27264 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
cf0ddf686eec42e5897a2ea24534acfb | 127.22.242.126:45965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.242.84:38033 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
305daa9a4d874d9db0e5378e5d27b617 | 127.22.242.66:38751 | HEALTHY | <none> | 3 | 0
4165ee79ee03466981e067d998339611 | 127.22.242.68:41205 | HEALTHY | <none> | 2 | 0
6b69a8c25e2b4de0bb10db8d7bdf83f0 | 127.22.242.71:42765 | HEALTHY | <none> | 0 | 0
820e18ddcc1e4d53821d43bc32652bc9 | 127.22.242.65:37969 | HEALTHY | <none> | 3 | 0
89a56aa776934c4582bfbdb0622b0cb3 | 127.22.242.70:44045 | HEALTHY | <none> | 1 | 0
91ec783c4d7f433d877ee3724dc20b49 | 127.22.242.72:41237 | HEALTHY | <none> | 1 | 0
cbb4bf53151a4061a81cf5bacf084516 | 127.22.242.69:39081 | HEALTHY | <none> | 2 | 0
f6c292faf79e49edabdbc7a310a30503 | 127.22.242.67:45423 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.22.242.65 | experimental | 127.22.242.65:37969
local_ip_for_outbound_sockets | 127.22.242.66 | experimental | 127.22.242.66:38751
local_ip_for_outbound_sockets | 127.22.242.67 | experimental | 127.22.242.67:45423
local_ip_for_outbound_sockets | 127.22.242.68 | experimental | 127.22.242.68:41205
local_ip_for_outbound_sockets | 127.22.242.69 | experimental | 127.22.242.69:39081
local_ip_for_outbound_sockets | 127.22.242.70 | experimental | 127.22.242.70:44045
local_ip_for_outbound_sockets | 127.22.242.71 | experimental | 127.22.242.71:42765
local_ip_for_outbound_sockets | 127.22.242.72 | experimental | 127.22.242.72:41237
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.242.65:37969
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.242.66:38751
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.242.67:45423
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.22.242.68:41205
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.242.69:39081
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.242.70:44045
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.22.242.71:42765
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.22.242.72:41237
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.242.84:38033 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:13:17.303459 23497 log_verifier.cc:126] Checking tablet 09ca4d63eab743b3977a47bcf120b3cf
I20240503 14:13:17.314680 23497 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 09ca4d63eab743b3977a47bcf120b3cf
I20240503 14:13:17.314780 23497 log_verifier.cc:126] Checking tablet 0c41cab275df431f8ca3df66f9dcb871
I20240503 14:13:17.450520 23497 log_verifier.cc:177] Verified matching terms for 333 ops in tablet 0c41cab275df431f8ca3df66f9dcb871
I20240503 14:13:17.450963 23497 log_verifier.cc:126] Checking tablet 27288fd1dcc8421dab42d3c282f718ae
I20240503 14:13:17.555636 23497 log_verifier.cc:177] Verified matching terms for 305 ops in tablet 27288fd1dcc8421dab42d3c282f718ae
I20240503 14:13:17.556035 23497 log_verifier.cc:126] Checking tablet 41b3bc788e164ce99c05a492ee768724
I20240503 14:13:17.669538 23497 log_verifier.cc:177] Verified matching terms for 328 ops in tablet 41b3bc788e164ce99c05a492ee768724
I20240503 14:13:17.669960 23497 log_verifier.cc:126] Checking tablet 4cca566ca06643ffbc72f02be6d544b4
I20240503 14:13:17.827479 23497 log_verifier.cc:177] Verified matching terms for 422 ops in tablet 4cca566ca06643ffbc72f02be6d544b4
I20240503 14:13:17.828033 23497 log_verifier.cc:126] Checking tablet 619d33242fee4c73b1abc8eafb1c00e8
I20240503 14:13:17.960362 23497 log_verifier.cc:177] Verified matching terms for 343 ops in tablet 619d33242fee4c73b1abc8eafb1c00e8
I20240503 14:13:17.960785 23497 log_verifier.cc:126] Checking tablet 6385b8e7ad5849c9a17136f0967c516f
I20240503 14:13:18.078284 23497 log_verifier.cc:177] Verified matching terms for 336 ops in tablet 6385b8e7ad5849c9a17136f0967c516f
I20240503 14:13:18.078701 23497 log_verifier.cc:126] Checking tablet 7efea07ad3204b6e868a05e9d80f5b3a
I20240503 14:13:18.191637 23497 log_verifier.cc:177] Verified matching terms for 317 ops in tablet 7efea07ad3204b6e868a05e9d80f5b3a
I20240503 14:13:18.192054 23497 log_verifier.cc:126] Checking tablet 927839713c994a9d9ad9f071a6943b0d
I20240503 14:13:18.320809 23497 log_verifier.cc:177] Verified matching terms for 355 ops in tablet 927839713c994a9d9ad9f071a6943b0d
I20240503 14:13:18.321321 23497 log_verifier.cc:126] Checking tablet ab63a1d4c9764427a141feb6614fe18f
I20240503 14:13:18.437096 23497 log_verifier.cc:177] Verified matching terms for 313 ops in tablet ab63a1d4c9764427a141feb6614fe18f
I20240503 14:13:18.437494 23497 log_verifier.cc:126] Checking tablet c47ed0ca358f4370b61529f8ffba2503
I20240503 14:13:18.593169 23497 log_verifier.cc:177] Verified matching terms for 447 ops in tablet c47ed0ca358f4370b61529f8ffba2503
I20240503 14:13:18.593719 23497 log_verifier.cc:126] Checking tablet c6d0c7755fb8449ea077c3786dc619fc
I20240503 14:13:18.722332 23497 log_verifier.cc:177] Verified matching terms for 374 ops in tablet c6d0c7755fb8449ea077c3786dc619fc
I20240503 14:13:18.722822 23497 log_verifier.cc:126] Checking tablet eee65dca344f41528758a1babd90a2ff
I20240503 14:13:18.842031 23497 log_verifier.cc:177] Verified matching terms for 340 ops in tablet eee65dca344f41528758a1babd90a2ff
I20240503 14:13:18.842463 23497 log_verifier.cc:126] Checking tablet efa780a04e854838a933be5934cd1578
I20240503 14:13:18.971244 23497 log_verifier.cc:177] Verified matching terms for 354 ops in tablet efa780a04e854838a933be5934cd1578
I20240503 14:13:18.971705 23497 log_verifier.cc:126] Checking tablet f24cc60aba81429ca7275571faf77cb9
I20240503 14:13:19.123658 23497 log_verifier.cc:177] Verified matching terms for 418 ops in tablet f24cc60aba81429ca7275571faf77cb9
I20240503 14:13:19.124220 23497 log_verifier.cc:126] Checking tablet f2db6b6ac8bb4fffb18e0b65dea27b5c
I20240503 14:13:19.249538 23497 log_verifier.cc:177] Verified matching terms for 362 ops in tablet f2db6b6ac8bb4fffb18e0b65dea27b5c
I20240503 14:13:19.256837 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 26941
I20240503 14:13:19.258076 27062 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:19.494498 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 26941
W20240503 14:13:19.538194 30013 connection.cc:535] server connection from 127.22.242.65:54949 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:13:19.538666 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 27072
I20240503 14:13:19.539556 27194 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:19.770153 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 27072
W20240503 14:13:19.810634 28669 connection.cc:535] server connection from 127.22.242.66:48787 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:13:19.811072 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 27202
I20240503 14:13:19.812098 27324 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:13:20.004539 28276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.242.66:38751: connect: Connection refused (error 111)
W20240503 14:13:20.006839 28276 consensus_peers.cc:485] T eee65dca344f41528758a1babd90a2ff P 4165ee79ee03466981e067d998339611 -> Peer 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751): Couldn't send request to peer 305daa9a4d874d9db0e5378e5d27b617. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.66:38751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:20.030936 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 27202
W20240503 14:13:20.070453 28669 connection.cc:535] server connection from 127.22.242.67:43337 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:20.070704 26867 connection.cc:535] server connection from 127.22.242.67:45805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:13:20.071158 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 28259
I20240503 14:13:20.081149 28381 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:20.248024 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 28259
I20240503 14:13:20.276055 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 28390
I20240503 14:13:20.281725 28512 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:20.440797 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 28390
I20240503 14:13:20.469635 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 28520
I20240503 14:13:20.475296 28644 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:20.634737 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 28520
W20240503 14:13:20.663022 28669 connection.cc:535] server connection from 127.22.242.70:51805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:13:20.663394 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 28654
I20240503 14:13:20.668941 28775 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:20.812424 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 28654
W20240503 14:13:20.839705 26869 connection.cc:535] server connection from 127.22.242.71:34653 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:13:20.839953 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 29994
I20240503 14:13:20.845566 30117 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:20.965341 30752 raft_consensus.cc:482] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 305daa9a4d874d9db0e5378e5d27b617)
I20240503 14:13:20.965564 30752 raft_consensus.cc:504] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "305daa9a4d874d9db0e5378e5d27b617" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 38751 } } peers { permanent_uuid: "91ec783c4d7f433d877ee3724dc20b49" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 41237 } attrs { promote: false } }
I20240503 14:13:20.966480 30752 leader_election.cc:288] T 619d33242fee4c73b1abc8eafb1c00e8 P 91ec783c4d7f433d877ee3724dc20b49 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 305daa9a4d874d9db0e5378e5d27b617 (127.22.242.66:38751)
W20240503 14:13:20.967034 30012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.242.70:44045: connect: Connection refused (error 111)
I20240503 14:13:20.967121 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 29994
I20240503 14:13:20.987553 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 26851
I20240503 14:13:20.988615 26911 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:21.150390 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 26851
2024-05-03T14:13:21Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/3 (50319 ms)
[----------] 1 test from RebalanceParamTest (50319 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240503 14:13:21.239043 23497 test_util.cc:268] Using random seed: 1383147510
I20240503 14:13:21.240793 23497 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:13:21.240844 23497 ts_itest-base.cc:113] --------------
I20240503 14:13:21.240865 23497 ts_itest-base.cc:114] 8 tablet servers
I20240503 14:13:21.240898 23497 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:13:21.240942 23497 ts_itest-base.cc:116] --------------
2024-05-03T14:13:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:13:21Z Disabled control of system clock
I20240503 14:13:21.283537 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.242.126:34247
--webserver_interface=127.22.242.126
--webserver_port=0
--builtin_ntp_servers=127.22.242.84:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.242.126:34247
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 14:13:21.611801 30769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:21.612134 30769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:21.612237 30769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:21.620437 30769 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:21.620589 30769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:13:21.620659 30769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:21.620743 30769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:13:21.620806 30769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:13:21.630172 30769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.242.126:34247
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.242.126:34247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.242.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:21.631547 30769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:21.633354 30769 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:21.643822 30774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:21.643960 30777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:21.643926 30775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:21.644892 30769 server_base.cc:1034] running on GCE node
I20240503 14:13:21.645598 30769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:21.647024 30769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:21.648236 30769 hybrid_clock.cc:648] HybridClock initialized: now 1714745601648188 us; error 39 us; skew 500 ppm
I20240503 14:13:21.648561 30769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:21.650689 30769 webserver.cc:460] Webserver started at http://127.22.242.126:43811/ using document root <none> and password file <none>
I20240503 14:13:21.651373 30769 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:21.651485 30769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:21.651790 30769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:21.654341 30769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "bc85f90e5f3d49e5891f2b622a74cfb9"
format_stamp: "Formatted at 2024-05-03 14:13:21 on dist-test-slave-r3pd"
I20240503 14:13:21.655086 30769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "bc85f90e5f3d49e5891f2b622a74cfb9"
format_stamp: "Formatted at 2024-05-03 14:13:21 on dist-test-slave-r3pd"
I20240503 14:13:21.660035 30769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20240503 14:13:21.663161 30783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:21.664413 30769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:21.664566 30769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
uuid: "bc85f90e5f3d49e5891f2b622a74cfb9"
format_stamp: "Formatted at 2024-05-03 14:13:21 on dist-test-slave-r3pd"
I20240503 14:13:21.664757 30769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:21.686729 30769 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:21.687548 30769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:21.687814 30769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:21.704692 30769 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.126:34247
I20240503 14:13:21.704711 30834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.126:34247 every 8 connection(s)
I20240503 14:13:21.706055 30769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:13:21.709736 30835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:21.710928 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 30769
I20240503 14:13:21.711093 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:13:21.721913 30835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9: Bootstrap starting.
I20240503 14:13:21.725719 30835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:21.726999 30835 log.cc:824] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:21.729372 30835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9: No bootstrap required, opened a new log
I20240503 14:13:21.735836 30835 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc85f90e5f3d49e5891f2b622a74cfb9" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 34247 } }
I20240503 14:13:21.736193 30835 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:21.736284 30835 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc85f90e5f3d49e5891f2b622a74cfb9, State: Initialized, Role: FOLLOWER
I20240503 14:13:21.736821 30835 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc85f90e5f3d49e5891f2b622a74cfb9" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 34247 } }
I20240503 14:13:21.736999 30835 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:13:21.737108 30835 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:13:21.737231 30835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.739395 30835 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc85f90e5f3d49e5891f2b622a74cfb9" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 34247 } }
I20240503 14:13:21.739840 30835 leader_election.cc:302] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc85f90e5f3d49e5891f2b622a74cfb9; no voters:
I20240503 14:13:21.740406 30835 leader_election.cc:288] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:13:21.740537 30840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:21.741122 30840 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9 [term 1 LEADER]: Becoming Leader. State: Replica: bc85f90e5f3d49e5891f2b622a74cfb9, State: Running, Role: LEADER
I20240503 14:13:21.741670 30840 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc85f90e5f3d49e5891f2b622a74cfb9" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 34247 } }
I20240503 14:13:21.742343 30835 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:13:21.744401 30842 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc85f90e5f3d49e5891f2b622a74cfb9. Latest consensus state: current_term: 1 leader_uuid: "bc85f90e5f3d49e5891f2b622a74cfb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc85f90e5f3d49e5891f2b622a74cfb9" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 34247 } } }
I20240503 14:13:21.744735 30842 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9 [sys.catalog]: This master's current role is: LEADER
I20240503 14:13:21.746647 30841 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc85f90e5f3d49e5891f2b622a74cfb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc85f90e5f3d49e5891f2b622a74cfb9" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 34247 } } }
I20240503 14:13:21.746937 30841 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9 [sys.catalog]: This master's current role is: LEADER
I20240503 14:13:21.747081 30848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:13:21.751521 30848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:13:21.759799 30848 catalog_manager.cc:1347] Generated new cluster ID: b3adf7a6947b4ec7b984a69b4de8dfb5
I20240503 14:13:21.759909 30848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:13:21.780588 30848 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:13:21.782068 30848 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:13:21.791554 30848 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bc85f90e5f3d49e5891f2b622a74cfb9: Generated new TSK 0
I20240503 14:13:21.792754 30848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:13:21.798480 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.65:0
--local_ip_for_outbound_sockets=127.22.242.65
--webserver_interface=127.22.242.65
--webserver_port=0
--tserver_master_addrs=127.22.242.126:34247
--builtin_ntp_servers=127.22.242.84:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:13:22.134042 30859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:22.134352 30859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:22.134440 30859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:22.141930 30859 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:22.142055 30859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:22.142228 30859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.65
I20240503 14:13:22.151324 30859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.242.65
--webserver_port=0
--tserver_master_addrs=127.22.242.126:34247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.65
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:22.152590 30859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:22.154184 30859 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:22.165102 30864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:22.165199 30867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:22.165230 30865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:22.166246 30859 server_base.cc:1034] running on GCE node
I20240503 14:13:22.166779 30859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:22.167608 30859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:22.168834 30859 hybrid_clock.cc:648] HybridClock initialized: now 1714745602168784 us; error 40 us; skew 500 ppm
I20240503 14:13:22.169150 30859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:22.171206 30859 webserver.cc:460] Webserver started at http://127.22.242.65:37773/ using document root <none> and password file <none>
I20240503 14:13:22.171869 30859 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:22.171974 30859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:22.172334 30859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:22.174724 30859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0dc902d91ccc435a82a5ede4a7729811"
format_stamp: "Formatted at 2024-05-03 14:13:22 on dist-test-slave-r3pd"
I20240503 14:13:22.175447 30859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0dc902d91ccc435a82a5ede4a7729811"
format_stamp: "Formatted at 2024-05-03 14:13:22 on dist-test-slave-r3pd"
I20240503 14:13:22.180090 30859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240503 14:13:22.183238 30873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:22.184501 30859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:22.184646 30859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0dc902d91ccc435a82a5ede4a7729811"
format_stamp: "Formatted at 2024-05-03 14:13:22 on dist-test-slave-r3pd"
I20240503 14:13:22.184823 30859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:22.201519 30859 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:22.202340 30859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:22.202601 30859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:22.203523 30859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:22.205179 30859 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:22.205286 30859 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:22.205394 30859 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:22.205466 30859 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:22.238090 30859 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.65:46013
I20240503 14:13:22.238162 30985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.65:46013 every 8 connection(s)
I20240503 14:13:22.239408 30859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:13:22.247381 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 30859
I20240503 14:13:22.247578 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:13:22.251016 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.66:0
--local_ip_for_outbound_sockets=127.22.242.66
--webserver_interface=127.22.242.66
--webserver_port=0
--tserver_master_addrs=127.22.242.126:34247
--builtin_ntp_servers=127.22.242.84:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:13:22.252882 30986 heartbeater.cc:347] Connected to a master server at 127.22.242.126:34247
I20240503 14:13:22.253229 30986 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:22.254240 30986 heartbeater.cc:511] Master 127.22.242.126:34247 requested a full tablet report, sending...
I20240503 14:13:22.256354 30800 ts_manager.cc:194] Registered new tserver with Master: 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:22.257759 30800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.65:53533
W20240503 14:13:22.582978 30990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:22.583288 30990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:22.583410 30990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:22.591141 30990 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:22.591275 30990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:22.591444 30990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.66
I20240503 14:13:22.600626 30990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.242.66
--webserver_port=0
--tserver_master_addrs=127.22.242.126:34247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.66
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:22.602010 30990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:22.603754 30990 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:22.615514 30998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:22.615628 30996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:22.615474 30995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:22.616534 30990 server_base.cc:1034] running on GCE node
I20240503 14:13:22.617141 30990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:22.618073 30990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:22.619316 30990 hybrid_clock.cc:648] HybridClock initialized: now 1714745602619264 us; error 42 us; skew 500 ppm
I20240503 14:13:22.619643 30990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:22.621855 30990 webserver.cc:460] Webserver started at http://127.22.242.66:45103/ using document root <none> and password file <none>
I20240503 14:13:22.622582 30990 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:22.622691 30990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:22.622995 30990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:22.625564 30990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6dc4948ed7c740ab89b9e349d151b0e5"
format_stamp: "Formatted at 2024-05-03 14:13:22 on dist-test-slave-r3pd"
I20240503 14:13:22.626307 30990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6dc4948ed7c740ab89b9e349d151b0e5"
format_stamp: "Formatted at 2024-05-03 14:13:22 on dist-test-slave-r3pd"
I20240503 14:13:22.631311 30990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20240503 14:13:22.634609 31004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:22.635866 30990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:22.636018 30990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6dc4948ed7c740ab89b9e349d151b0e5"
format_stamp: "Formatted at 2024-05-03 14:13:22 on dist-test-slave-r3pd"
I20240503 14:13:22.636248 30990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:22.653925 30990 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:22.654748 30990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:22.655033 30990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:22.655951 30990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:22.657564 30990 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:22.657675 30990 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:22.657789 30990 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:22.657860 30990 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:22.688133 30990 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.66:33221
I20240503 14:13:22.688211 31116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.66:33221 every 8 connection(s)
I20240503 14:13:22.689539 30990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:13:22.689953 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 30990
I20240503 14:13:22.690143 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:13:22.693677 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.67:0
--local_ip_for_outbound_sockets=127.22.242.67
--webserver_interface=127.22.242.67
--webserver_port=0
--tserver_master_addrs=127.22.242.126:34247
--builtin_ntp_servers=127.22.242.84:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:13:22.702176 31117 heartbeater.cc:347] Connected to a master server at 127.22.242.126:34247
I20240503 14:13:22.702476 31117 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:22.703295 31117 heartbeater.cc:511] Master 127.22.242.126:34247 requested a full tablet report, sending...
I20240503 14:13:22.704879 30800 ts_manager.cc:194] Registered new tserver with Master: 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:22.705976 30800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.66:53961
W20240503 14:13:23.023672 31120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:23.023958 31120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:23.024055 31120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:23.031360 31120 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:23.031484 31120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:23.031605 31120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.67
I20240503 14:13:23.040664 31120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.242.67
--webserver_port=0
--tserver_master_addrs=127.22.242.126:34247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.67
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:23.042043 31120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:23.043835 31120 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:23.055210 31125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:23.055292 31126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:23.055327 31128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:23.056252 31120 server_base.cc:1034] running on GCE node
I20240503 14:13:23.056820 31120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:23.057694 31120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:23.058888 31120 hybrid_clock.cc:648] HybridClock initialized: now 1714745603058839 us; error 46 us; skew 500 ppm
I20240503 14:13:23.059173 31120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:23.061272 31120 webserver.cc:460] Webserver started at http://127.22.242.67:45845/ using document root <none> and password file <none>
I20240503 14:13:23.061954 31120 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:23.062062 31120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:23.062379 31120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:23.064818 31120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce"
format_stamp: "Formatted at 2024-05-03 14:13:23 on dist-test-slave-r3pd"
I20240503 14:13:23.065583 31120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce"
format_stamp: "Formatted at 2024-05-03 14:13:23 on dist-test-slave-r3pd"
I20240503 14:13:23.070359 31120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:13:23.073375 31134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:23.074591 31120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240503 14:13:23.074759 31120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce"
format_stamp: "Formatted at 2024-05-03 14:13:23 on dist-test-slave-r3pd"
I20240503 14:13:23.074939 31120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:23.096896 31120 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:23.097746 31120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:23.098012 31120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:23.098930 31120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:23.100368 31120 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:23.100466 31120 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:23.100569 31120 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:23.100646 31120 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:23.131425 31120 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.67:44297
I20240503 14:13:23.131489 31246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.67:44297 every 8 connection(s)
I20240503 14:13:23.132848 31120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:13:23.141923 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 31120
I20240503 14:13:23.142082 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:13:23.145022 31247 heartbeater.cc:347] Connected to a master server at 127.22.242.126:34247
I20240503 14:13:23.145308 31247 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:23.145391 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.68:0
--local_ip_for_outbound_sockets=127.22.242.68
--webserver_interface=127.22.242.68
--webserver_port=0
--tserver_master_addrs=127.22.242.126:34247
--builtin_ntp_servers=127.22.242.84:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:13:23.146140 31247 heartbeater.cc:511] Master 127.22.242.126:34247 requested a full tablet report, sending...
I20240503 14:13:23.147526 30800 ts_manager.cc:194] Registered new tserver with Master: e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:23.148329 30800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.67:59969
I20240503 14:13:23.261034 30986 heartbeater.cc:503] Master 127.22.242.126:34247 was elected leader, sending a full tablet report...
W20240503 14:13:23.480751 31250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:23.481052 31250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:23.481189 31250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:23.488575 31250 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:23.488713 31250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:23.488873 31250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.68
I20240503 14:13:23.497530 31250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.242.68
--webserver_port=0
--tserver_master_addrs=127.22.242.126:34247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.68
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:23.498860 31250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:23.500536 31250 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:23.511976 31256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:23.511996 31258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:23.512220 31255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:23.512768 31250 server_base.cc:1034] running on GCE node
I20240503 14:13:23.513355 31250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:23.514273 31250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:23.515470 31250 hybrid_clock.cc:648] HybridClock initialized: now 1714745603515420 us; error 47 us; skew 500 ppm
I20240503 14:13:23.515784 31250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:23.518143 31250 webserver.cc:460] Webserver started at http://127.22.242.68:37045/ using document root <none> and password file <none>
I20240503 14:13:23.519061 31250 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:23.519177 31250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:23.519500 31250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:23.522111 31250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "82cfdc130877418dbe9d980d368973be"
format_stamp: "Formatted at 2024-05-03 14:13:23 on dist-test-slave-r3pd"
I20240503 14:13:23.522884 31250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "82cfdc130877418dbe9d980d368973be"
format_stamp: "Formatted at 2024-05-03 14:13:23 on dist-test-slave-r3pd"
I20240503 14:13:23.527982 31250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20240503 14:13:23.531347 31264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:23.532656 31250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:23.532829 31250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "82cfdc130877418dbe9d980d368973be"
format_stamp: "Formatted at 2024-05-03 14:13:23 on dist-test-slave-r3pd"
I20240503 14:13:23.533028 31250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:23.553352 31250 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:23.554170 31250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:23.554423 31250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:23.555353 31250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:23.556841 31250 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:23.556941 31250 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:23.557060 31250 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:23.557132 31250 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:23.588368 31250 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.68:37063
I20240503 14:13:23.588440 31376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.68:37063 every 8 connection(s)
I20240503 14:13:23.589800 31250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:13:23.593266 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 31250
I20240503 14:13:23.593458 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:13:23.596915 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.69:0
--local_ip_for_outbound_sockets=127.22.242.69
--webserver_interface=127.22.242.69
--webserver_port=0
--tserver_master_addrs=127.22.242.126:34247
--builtin_ntp_servers=127.22.242.84:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:13:23.603886 31377 heartbeater.cc:347] Connected to a master server at 127.22.242.126:34247
I20240503 14:13:23.604269 31377 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:23.605123 31377 heartbeater.cc:511] Master 127.22.242.126:34247 requested a full tablet report, sending...
I20240503 14:13:23.606568 30800 ts_manager.cc:194] Registered new tserver with Master: 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:23.607491 30800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.68:42781
I20240503 14:13:23.709167 31117 heartbeater.cc:503] Master 127.22.242.126:34247 was elected leader, sending a full tablet report...
W20240503 14:13:23.932190 31380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:23.932518 31380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:23.932653 31380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:23.940411 31380 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:23.940559 31380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:23.940712 31380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.69
I20240503 14:13:23.949993 31380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.242.69
--webserver_port=0
--tserver_master_addrs=127.22.242.126:34247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.69
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:23.951320 31380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:23.953089 31380 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:23.963965 31386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:23.963972 31385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:23.965510 31388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:23.965631 31380 server_base.cc:1034] running on GCE node
I20240503 14:13:23.966224 31380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:23.967061 31380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:23.968230 31380 hybrid_clock.cc:648] HybridClock initialized: now 1714745603968178 us; error 44 us; skew 500 ppm
I20240503 14:13:23.968566 31380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:23.970649 31380 webserver.cc:460] Webserver started at http://127.22.242.69:44473/ using document root <none> and password file <none>
I20240503 14:13:23.971340 31380 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:23.971448 31380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:23.971740 31380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:23.974241 31380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "6967bdda50404d68a5ac043b709440e1"
format_stamp: "Formatted at 2024-05-03 14:13:23 on dist-test-slave-r3pd"
I20240503 14:13:23.974949 31380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "6967bdda50404d68a5ac043b709440e1"
format_stamp: "Formatted at 2024-05-03 14:13:23 on dist-test-slave-r3pd"
I20240503 14:13:23.979887 31380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240503 14:13:23.983222 31394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:23.984544 31380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240503 14:13:23.984723 31380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6967bdda50404d68a5ac043b709440e1"
format_stamp: "Formatted at 2024-05-03 14:13:23 on dist-test-slave-r3pd"
I20240503 14:13:23.984906 31380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:24.005676 31380 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:24.006552 31380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:24.006848 31380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:24.007800 31380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:24.009373 31380 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:24.009476 31380 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:24.009595 31380 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:24.009668 31380 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:24.042011 31380 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.69:42189
I20240503 14:13:24.042070 31506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.69:42189 every 8 connection(s)
I20240503 14:13:24.043363 31380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:13:24.047336 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 31380
I20240503 14:13:24.047508 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:13:24.050834 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.70:0
--local_ip_for_outbound_sockets=127.22.242.70
--webserver_interface=127.22.242.70
--webserver_port=0
--tserver_master_addrs=127.22.242.126:34247
--builtin_ntp_servers=127.22.242.84:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:13:24.056818 31507 heartbeater.cc:347] Connected to a master server at 127.22.242.126:34247
I20240503 14:13:24.057094 31507 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:24.057883 31507 heartbeater.cc:511] Master 127.22.242.126:34247 requested a full tablet report, sending...
I20240503 14:13:24.059180 30800 ts_manager.cc:194] Registered new tserver with Master: 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189)
I20240503 14:13:24.059891 30800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.69:46783
I20240503 14:13:24.150902 31247 heartbeater.cc:503] Master 127.22.242.126:34247 was elected leader, sending a full tablet report...
W20240503 14:13:24.382833 31510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:24.383128 31510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:24.383263 31510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:24.390815 31510 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:24.390933 31510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:24.391088 31510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.70
I20240503 14:13:24.400118 31510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.242.70
--webserver_port=0
--tserver_master_addrs=127.22.242.126:34247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.70
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:24.401482 31510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:24.403179 31510 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:24.414948 31516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:24.414959 31515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:24.415179 31518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:24.415894 31510 server_base.cc:1034] running on GCE node
I20240503 14:13:24.416543 31510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:24.417508 31510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:24.418710 31510 hybrid_clock.cc:648] HybridClock initialized: now 1714745604418665 us; error 39 us; skew 500 ppm
I20240503 14:13:24.419034 31510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:24.421293 31510 webserver.cc:460] Webserver started at http://127.22.242.70:45503/ using document root <none> and password file <none>
I20240503 14:13:24.421984 31510 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:24.422084 31510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:24.422426 31510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:24.425029 31510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "fa71cbdf0e544b81bd4e7e27196d2c91"
format_stamp: "Formatted at 2024-05-03 14:13:24 on dist-test-slave-r3pd"
I20240503 14:13:24.425897 31510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "fa71cbdf0e544b81bd4e7e27196d2c91"
format_stamp: "Formatted at 2024-05-03 14:13:24 on dist-test-slave-r3pd"
I20240503 14:13:24.430941 31510 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:13:24.434232 31524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:24.435606 31510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:24.435773 31510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "fa71cbdf0e544b81bd4e7e27196d2c91"
format_stamp: "Formatted at 2024-05-03 14:13:24 on dist-test-slave-r3pd"
I20240503 14:13:24.435971 31510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:24.467432 31510 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:24.468317 31510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:24.468575 31510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:24.469564 31510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:24.470999 31510 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:24.471096 31510 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:24.471202 31510 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:24.471271 31510 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:24.501763 31510 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.70:46829
I20240503 14:13:24.501844 31636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.70:46829 every 8 connection(s)
I20240503 14:13:24.503122 31510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:13:24.509897 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 31510
I20240503 14:13:24.510032 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 14:13:24.512955 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.71:0
--local_ip_for_outbound_sockets=127.22.242.71
--webserver_interface=127.22.242.71
--webserver_port=0
--tserver_master_addrs=127.22.242.126:34247
--builtin_ntp_servers=127.22.242.84:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:13:24.515741 31637 heartbeater.cc:347] Connected to a master server at 127.22.242.126:34247
I20240503 14:13:24.516041 31637 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:24.517027 31637 heartbeater.cc:511] Master 127.22.242.126:34247 requested a full tablet report, sending...
I20240503 14:13:24.518440 30800 ts_manager.cc:194] Registered new tserver with Master: fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829)
I20240503 14:13:24.519165 30800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.70:58487
I20240503 14:13:24.609989 31377 heartbeater.cc:503] Master 127.22.242.126:34247 was elected leader, sending a full tablet report...
W20240503 14:13:24.850093 31640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:24.850361 31640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:24.850498 31640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:24.857767 31640 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:24.857887 31640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:24.858060 31640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.71
I20240503 14:13:24.866786 31640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.242.71
--webserver_port=0
--tserver_master_addrs=127.22.242.126:34247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.71
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:24.868109 31640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:24.869843 31640 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:24.881062 31648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:24.881008 31645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:24.880980 31646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:24.882050 31640 server_base.cc:1034] running on GCE node
I20240503 14:13:24.882612 31640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:24.883549 31640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:24.884737 31640 hybrid_clock.cc:648] HybridClock initialized: now 1714745604884691 us; error 45 us; skew 500 ppm
I20240503 14:13:24.885032 31640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:24.887187 31640 webserver.cc:460] Webserver started at http://127.22.242.71:43355/ using document root <none> and password file <none>
I20240503 14:13:24.887904 31640 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:24.888027 31640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:24.888384 31640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:24.891045 31640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "4665a3377574417ba0d3165c236533c0"
format_stamp: "Formatted at 2024-05-03 14:13:24 on dist-test-slave-r3pd"
I20240503 14:13:24.891788 31640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "4665a3377574417ba0d3165c236533c0"
format_stamp: "Formatted at 2024-05-03 14:13:24 on dist-test-slave-r3pd"
I20240503 14:13:24.896764 31640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20240503 14:13:24.899997 31654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:24.901379 31640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:24.901525 31640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4665a3377574417ba0d3165c236533c0"
format_stamp: "Formatted at 2024-05-03 14:13:24 on dist-test-slave-r3pd"
I20240503 14:13:24.901727 31640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:24.930944 31640 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:24.931860 31640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:24.932112 31640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:24.933039 31640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:24.934456 31640 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:24.934554 31640 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:24.934667 31640 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:24.934738 31640 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:24.966135 31640 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.71:46453
I20240503 14:13:24.966238 31766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.71:46453 every 8 connection(s)
I20240503 14:13:24.967677 31640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:13:24.972534 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 31640
I20240503 14:13:24.972668 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 14:13:24.975824 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.72:0
--local_ip_for_outbound_sockets=127.22.242.72
--webserver_interface=127.22.242.72
--webserver_port=0
--tserver_master_addrs=127.22.242.126:34247
--builtin_ntp_servers=127.22.242.84:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:13:24.980429 31767 heartbeater.cc:347] Connected to a master server at 127.22.242.126:34247
I20240503 14:13:24.980719 31767 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:24.981506 31767 heartbeater.cc:511] Master 127.22.242.126:34247 requested a full tablet report, sending...
I20240503 14:13:24.982795 30800 ts_manager.cc:194] Registered new tserver with Master: 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453)
I20240503 14:13:24.983551 30800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.71:35725
I20240503 14:13:25.062314 31507 heartbeater.cc:503] Master 127.22.242.126:34247 was elected leader, sending a full tablet report...
W20240503 14:13:25.314392 31770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:25.314700 31770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:25.314829 31770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:25.322824 31770 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:25.322952 31770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:25.323122 31770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.72
I20240503 14:13:25.332258 31770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.242.72
--webserver_port=0
--tserver_master_addrs=127.22.242.126:34247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.72
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:25.333626 31770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:25.335327 31770 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:25.347429 31775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:25.347884 31778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:25.347452 31776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:25.348320 31770 server_base.cc:1034] running on GCE node
I20240503 14:13:25.348919 31770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:25.349857 31770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:25.351104 31770 hybrid_clock.cc:648] HybridClock initialized: now 1714745605351031 us; error 49 us; skew 500 ppm
I20240503 14:13:25.351410 31770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:25.359201 31770 webserver.cc:460] Webserver started at http://127.22.242.72:44281/ using document root <none> and password file <none>
I20240503 14:13:25.359933 31770 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:25.360049 31770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:25.360417 31770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:25.362987 31770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "85f69eefe4294197923dff5efadcbbd5"
format_stamp: "Formatted at 2024-05-03 14:13:25 on dist-test-slave-r3pd"
I20240503 14:13:25.363762 31770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "85f69eefe4294197923dff5efadcbbd5"
format_stamp: "Formatted at 2024-05-03 14:13:25 on dist-test-slave-r3pd"
I20240503 14:13:25.368858 31770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 14:13:25.372345 31784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:25.373677 31770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:13:25.373836 31770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "85f69eefe4294197923dff5efadcbbd5"
format_stamp: "Formatted at 2024-05-03 14:13:25 on dist-test-slave-r3pd"
I20240503 14:13:25.374024 31770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:25.396589 31770 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:25.397439 31770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:25.397707 31770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:25.398662 31770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:25.400135 31770 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:25.400286 31770 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:25.400394 31770 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:25.400471 31770 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:25.432945 31770 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.72:36545
I20240503 14:13:25.432998 31896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.72:36545 every 8 connection(s)
I20240503 14:13:25.434316 31770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:13:25.434744 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 31770
I20240503 14:13:25.434895 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 14:13:25.449054 31897 heartbeater.cc:347] Connected to a master server at 127.22.242.126:34247
I20240503 14:13:25.449370 31897 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:25.450368 31897 heartbeater.cc:511] Master 127.22.242.126:34247 requested a full tablet report, sending...
I20240503 14:13:25.451790 30800 ts_manager.cc:194] Registered new tserver with Master: 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545)
I20240503 14:13:25.452589 30800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.72:49001
I20240503 14:13:25.463557 23497 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 14:13:25.479696 23497 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 14:13:25.479837 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 31380
I20240503 14:13:25.487578 31502 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:25.521782 31637 heartbeater.cc:503] Master 127.22.242.126:34247 was elected leader, sending a full tablet report...
I20240503 14:13:25.569664 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 31380
I20240503 14:13:25.586326 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 31510
I20240503 14:13:25.594708 31632 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:25.675827 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 31510
I20240503 14:13:25.693564 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 31640
I20240503 14:13:25.701269 31762 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:25.789363 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 31640
I20240503 14:13:25.805645 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 31770
I20240503 14:13:25.814152 31892 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:25.895749 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 31770
I20240503 14:13:29.663902 30800 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36072:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:29.730938 30921 tablet_service.cc:1467] Processing CreateTablet for tablet 2f82073122a74268a464387659dd90bf (DEFAULT_TABLE table=rebalance_test_table_0 [id=fa8e781fa3e140b6865ce5f7b429e4c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.732281 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 63b2120788a54af78f02449f46bf23c7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fa8e781fa3e140b6865ce5f7b429e4c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.732535 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 5abcdc783c6144f28b9f8a9e1e162006 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fa8e781fa3e140b6865ce5f7b429e4c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.733646 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f82073122a74268a464387659dd90bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.734215 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b2120788a54af78f02449f46bf23c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.735654 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5abcdc783c6144f28b9f8a9e1e162006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.736189 31312 tablet_service.cc:1467] Processing CreateTablet for tablet 2f82073122a74268a464387659dd90bf (DEFAULT_TABLE table=rebalance_test_table_0 [id=fa8e781fa3e140b6865ce5f7b429e4c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.737641 31312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f82073122a74268a464387659dd90bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.738058 31311 tablet_service.cc:1467] Processing CreateTablet for tablet 63b2120788a54af78f02449f46bf23c7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fa8e781fa3e140b6865ce5f7b429e4c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.740082 31311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b2120788a54af78f02449f46bf23c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.741878 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 5abcdc783c6144f28b9f8a9e1e162006 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fa8e781fa3e140b6865ce5f7b429e4c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.743292 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5abcdc783c6144f28b9f8a9e1e162006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.751271 31182 tablet_service.cc:1467] Processing CreateTablet for tablet 2f82073122a74268a464387659dd90bf (DEFAULT_TABLE table=rebalance_test_table_0 [id=fa8e781fa3e140b6865ce5f7b429e4c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.751732 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f82073122a74268a464387659dd90bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.752426 31052 tablet_service.cc:1467] Processing CreateTablet for tablet 5abcdc783c6144f28b9f8a9e1e162006 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fa8e781fa3e140b6865ce5f7b429e4c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.752465 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 63b2120788a54af78f02449f46bf23c7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fa8e781fa3e140b6865ce5f7b429e4c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.753855 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5abcdc783c6144f28b9f8a9e1e162006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.754279 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b2120788a54af78f02449f46bf23c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.773380 31928 tablet_bootstrap.cc:492] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:29.774917 31927 tablet_bootstrap.cc:492] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:29.778357 31928 tablet_bootstrap.cc:654] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.779834 31928 log.cc:824] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:29.780433 31927 tablet_bootstrap.cc:654] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.781111 31929 tablet_bootstrap.cc:492] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:29.782202 31927 log.cc:824] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:29.782874 31926 tablet_bootstrap.cc:492] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:29.784736 31927 tablet_bootstrap.cc:492] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:29.784996 31927 ts_tablet_manager.cc:1367] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240503 14:13:29.785238 31929 tablet_bootstrap.cc:654] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.786340 31926 tablet_bootstrap.cc:654] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.786449 31929 log.cc:824] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:29.787633 31926 log.cc:824] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:29.791972 31927 raft_consensus.cc:348] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:29.792451 31928 tablet_bootstrap.cc:492] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:29.792446 31929 tablet_bootstrap.cc:492] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:29.792563 31927 raft_consensus.cc:374] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.792673 31927 raft_consensus.cc:729] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.792721 31929 ts_tablet_manager.cc:1367] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20240503 14:13:29.792721 31928 ts_tablet_manager.cc:1367] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.000s
I20240503 14:13:29.793367 31927 consensus_queue.cc:260] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:29.794276 31926 tablet_bootstrap.cc:492] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:29.794586 31926 ts_tablet_manager.cc:1367] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240503 14:13:29.801568 31927 ts_tablet_manager.cc:1398] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20240503 14:13:29.801707 31929 raft_consensus.cc:348] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:29.802178 31927 tablet_bootstrap.cc:492] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:29.802245 31929 raft_consensus.cc:374] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.802434 31929 raft_consensus.cc:729] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.803203 31929 consensus_queue.cc:260] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:29.803973 31927 tablet_bootstrap.cc:654] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.803900 31928 raft_consensus.cc:348] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:29.804569 31928 raft_consensus.cc:374] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.804735 31928 raft_consensus.cc:729] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.808847 31928 consensus_queue.cc:260] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:29.808697 31926 raft_consensus.cc:348] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:29.809387 31926 raft_consensus.cc:374] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.809549 31926 raft_consensus.cc:729] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.810526 31926 consensus_queue.cc:260] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:29.812409 31929 ts_tablet_manager.cc:1398] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.020s user 0.016s sys 0.001s
I20240503 14:13:29.812899 31929 tablet_bootstrap.cc:492] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:29.814108 31927 tablet_bootstrap.cc:492] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:29.814394 31927 ts_tablet_manager.cc:1367] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20240503 14:13:29.814714 31929 tablet_bootstrap.cc:654] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.816143 31927 raft_consensus.cc:348] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:29.816578 31927 raft_consensus.cc:374] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.816745 31927 raft_consensus.cc:729] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.817049 31928 ts_tablet_manager.cc:1398] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.024s user 0.013s sys 0.000s
I20240503 14:13:29.817715 31928 tablet_bootstrap.cc:492] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:29.820250 31928 tablet_bootstrap.cc:654] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.817195 31927 consensus_queue.cc:260] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:29.821184 31927 ts_tablet_manager.cc:1398] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:13:29.823012 31926 ts_tablet_manager.cc:1398] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.028s user 0.013s sys 0.013s
I20240503 14:13:29.823310 31929 tablet_bootstrap.cc:492] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:29.823506 31929 ts_tablet_manager.cc:1367] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240503 14:13:29.823565 31926 tablet_bootstrap.cc:492] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:29.824626 31929 raft_consensus.cc:348] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:29.824852 31929 raft_consensus.cc:374] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.824970 31929 raft_consensus.cc:729] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.825210 31929 consensus_queue.cc:260] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:29.825650 31926 tablet_bootstrap.cc:654] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.825793 31929 ts_tablet_manager.cc:1398] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:29.829881 31928 tablet_bootstrap.cc:492] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:29.829881 31926 tablet_bootstrap.cc:492] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:29.830039 31926 ts_tablet_manager.cc:1367] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240503 14:13:29.830081 31928 ts_tablet_manager.cc:1367] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20240503 14:13:29.831073 31928 raft_consensus.cc:348] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:29.831072 31926 raft_consensus.cc:348] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:29.831290 31926 raft_consensus.cc:374] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.831290 31928 raft_consensus.cc:374] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.831388 31926 raft_consensus.cc:729] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.831388 31928 raft_consensus.cc:729] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.831627 31926 consensus_queue.cc:260] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:29.831627 31928 consensus_queue.cc:260] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:29.832150 31928 ts_tablet_manager.cc:1398] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:29.832691 31926 ts_tablet_manager.cc:1398] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:29.833496 31926 tablet_bootstrap.cc:492] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:29.835054 31926 tablet_bootstrap.cc:654] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.836792 31926 tablet_bootstrap.cc:492] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:29.836927 31926 ts_tablet_manager.cc:1367] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:13:29.837469 31926 raft_consensus.cc:348] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:29.837646 31926 raft_consensus.cc:374] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.837723 31926 raft_consensus.cc:729] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.837965 31926 consensus_queue.cc:260] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:29.838443 31926 ts_tablet_manager.cc:1398] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 14:13:29.845510 31378 tablet.cc:2369] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:29.854940 31934 raft_consensus.cc:482] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:29.855193 31934 raft_consensus.cc:504] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:29.857123 31934 leader_election.cc:288] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:29.864200 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5abcdc783c6144f28b9f8a9e1e162006" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" is_pre_election: true
I20240503 14:13:29.864193 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5abcdc783c6144f28b9f8a9e1e162006" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:29.864753 31202 raft_consensus.cc:2454] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:29.864753 30941 raft_consensus.cc:2454] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:29.865366 31005 leader_election.cc:302] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 6dc4948ed7c740ab89b9e349d151b0e5; no voters:
I20240503 14:13:29.865723 31934 raft_consensus.cc:2781] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:29.865860 31934 raft_consensus.cc:482] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:29.865947 31934 raft_consensus.cc:3037] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.867898 31934 raft_consensus.cc:504] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:29.868543 31934 leader_election.cc:288] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Requested vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:29.868932 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5abcdc783c6144f28b9f8a9e1e162006" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce"
I20240503 14:13:29.868991 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5abcdc783c6144f28b9f8a9e1e162006" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:29.869133 31202 raft_consensus.cc:3037] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.869150 30941 raft_consensus.cc:3037] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.871177 31202 raft_consensus.cc:2454] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:29.871188 30941 raft_consensus.cc:2454] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:29.871519 31005 leader_election.cc:302] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 6dc4948ed7c740ab89b9e349d151b0e5; no voters:
I20240503 14:13:29.871791 31934 raft_consensus.cc:2781] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:29.872256 31934 raft_consensus.cc:686] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Becoming Leader. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Running, Role: LEADER
I20240503 14:13:29.872684 31934 consensus_queue.cc:237] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:29.877241 30799 catalog_manager.cc:5484] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } }
W20240503 14:13:29.889449 31248 tablet.cc:2369] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:29.908192 31937 raft_consensus.cc:482] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:29.908388 31937 raft_consensus.cc:504] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:29.910084 31937 leader_election.cc:288] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:29.916635 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f82073122a74268a464387659dd90bf" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:29.916880 30941 raft_consensus.cc:2454] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 0.
I20240503 14:13:29.916924 31332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f82073122a74268a464387659dd90bf" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be" is_pre_election: true
I20240503 14:13:29.917341 31135 leader_election.cc:302] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:29.917443 31332 raft_consensus.cc:2454] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 0.
I20240503 14:13:29.917706 31937 raft_consensus.cc:2781] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:29.917845 31937 raft_consensus.cc:482] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:29.917951 31937 raft_consensus.cc:3037] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.919991 31937 raft_consensus.cc:504] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:29.920589 31937 leader_election.cc:288] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 election: Requested vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:29.921082 31332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f82073122a74268a464387659dd90bf" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be"
I20240503 14:13:29.921090 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f82073122a74268a464387659dd90bf" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:29.921289 30941 raft_consensus.cc:3037] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.921272 31332 raft_consensus.cc:3037] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.923101 30941 raft_consensus.cc:2454] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 1.
I20240503 14:13:29.923242 31332 raft_consensus.cc:2454] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 1.
I20240503 14:13:29.923476 31135 leader_election.cc:302] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:29.923748 31937 raft_consensus.cc:2781] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:29.924201 31937 raft_consensus.cc:686] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Becoming Leader. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Running, Role: LEADER
I20240503 14:13:29.924634 31937 consensus_queue.cc:237] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:29.928944 30799 catalog_manager.cc:5484] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: term changed from 0 to 1, leader changed from <none> to e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67). New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } }
W20240503 14:13:29.946111 31118 tablet.cc:2369] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:13:29.996094 30987 tablet.cc:2369] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:30.117384 31934 raft_consensus.cc:482] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:30.117684 31934 raft_consensus.cc:504] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:30.118443 31934 leader_election.cc:288] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:30.119134 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b2120788a54af78f02449f46bf23c7" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:30.119405 30941 raft_consensus.cc:2454] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:30.119868 31005 leader_election.cc:302] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 6dc4948ed7c740ab89b9e349d151b0e5; no voters:
I20240503 14:13:30.120143 31934 raft_consensus.cc:2781] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:30.120311 31934 raft_consensus.cc:482] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:30.120432 31934 raft_consensus.cc:3037] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.122777 31934 raft_consensus.cc:504] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:30.123528 31934 leader_election.cc:288] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Requested vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:30.123843 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b2120788a54af78f02449f46bf23c7" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:30.124001 30941 raft_consensus.cc:3037] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.124796 31332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b2120788a54af78f02449f46bf23c7" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be" is_pre_election: true
I20240503 14:13:30.124877 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b2120788a54af78f02449f46bf23c7" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be"
I20240503 14:13:30.125087 31332 raft_consensus.cc:2454] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:30.126420 30941 raft_consensus.cc:2454] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:30.126775 31005 leader_election.cc:302] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 6dc4948ed7c740ab89b9e349d151b0e5; no voters: 82cfdc130877418dbe9d980d368973be
I20240503 14:13:30.127132 31934 raft_consensus.cc:2781] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:30.127314 31934 raft_consensus.cc:686] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Becoming Leader. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Running, Role: LEADER
I20240503 14:13:30.127557 31934 consensus_queue.cc:237] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:30.129949 30799 catalog_manager.cc:5484] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:30.257539 31934 consensus_queue.cc:1035] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:30.266247 31945 consensus_queue.cc:1035] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:30.420248 31937 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:30.426057 31937 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:30.491209 31952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:30.491547 31952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:30.499110 31952 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 14:13:30.671149 30941 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:30.679144 31945 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:30.727844 31332 raft_consensus.cc:3037] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.731007 31332 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:30.731750 31934 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:30.766497 31962 mvcc.cc:205] Tried to move back new op lower bound from 7023598021303128064 to 7023598019084455936. Current Snapshot: MvccSnapshot[applied={T|T < 7023598021303128064}]
I20240503 14:13:30.770643 31989 mvcc.cc:205] Tried to move back new op lower bound from 7023598021303128064 to 7023598019084455936. Current Snapshot: MvccSnapshot[applied={T|T < 7023598021303128064}]
I20240503 14:13:30.776300 31956 mvcc.cc:205] Tried to move back new op lower bound from 7023598021303128064 to 7023598019084455936. Current Snapshot: MvccSnapshot[applied={T|T < 7023598021303128064}]
INSERT report
rows total: 2000
time total: 261.074 ms
time per row: 0.130537 ms
I20240503 14:13:30.909528 30800 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36072:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:30.918103 31312 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb75d7d9499475ca60c633c0801d639 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f6d48b36155b41d3a75fe51d7d204ea7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:30.918596 31312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb75d7d9499475ca60c633c0801d639. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.918525 31052 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb75d7d9499475ca60c633c0801d639 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f6d48b36155b41d3a75fe51d7d204ea7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:30.918941 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb75d7d9499475ca60c633c0801d639. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.919034 31182 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb75d7d9499475ca60c633c0801d639 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f6d48b36155b41d3a75fe51d7d204ea7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:30.919423 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb75d7d9499475ca60c633c0801d639. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.919776 30921 tablet_service.cc:1467] Processing CreateTablet for tablet ebd6dc710e144e61a5fae3911f25ee5c (DEFAULT_TABLE table=rebalance_test_table_1 [id=f6d48b36155b41d3a75fe51d7d204ea7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:30.920183 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd6dc710e144e61a5fae3911f25ee5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.922453 31051 tablet_service.cc:1467] Processing CreateTablet for tablet e7b5dd5eb46a45989bf9ffe3144bc189 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f6d48b36155b41d3a75fe51d7d204ea7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:30.922832 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b5dd5eb46a45989bf9ffe3144bc189. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.922911 31180 tablet_service.cc:1467] Processing CreateTablet for tablet ebd6dc710e144e61a5fae3911f25ee5c (DEFAULT_TABLE table=rebalance_test_table_1 [id=f6d48b36155b41d3a75fe51d7d204ea7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:30.923357 31180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd6dc710e144e61a5fae3911f25ee5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.927222 31050 tablet_service.cc:1467] Processing CreateTablet for tablet ebd6dc710e144e61a5fae3911f25ee5c (DEFAULT_TABLE table=rebalance_test_table_1 [id=f6d48b36155b41d3a75fe51d7d204ea7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:30.927635 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd6dc710e144e61a5fae3911f25ee5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.928551 31181 tablet_service.cc:1467] Processing CreateTablet for tablet e7b5dd5eb46a45989bf9ffe3144bc189 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f6d48b36155b41d3a75fe51d7d204ea7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:30.928951 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b5dd5eb46a45989bf9ffe3144bc189. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.932919 31311 tablet_service.cc:1467] Processing CreateTablet for tablet e7b5dd5eb46a45989bf9ffe3144bc189 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f6d48b36155b41d3a75fe51d7d204ea7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:30.933349 31311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b5dd5eb46a45989bf9ffe3144bc189. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.938900 32012 tablet_bootstrap.cc:492] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:30.942786 32012 tablet_bootstrap.cc:654] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.943972 32013 tablet_bootstrap.cc:492] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:30.944252 32011 tablet_bootstrap.cc:492] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:30.945943 32013 tablet_bootstrap.cc:654] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.946260 32012 tablet_bootstrap.cc:492] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:30.946359 32010 tablet_bootstrap.cc:492] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:30.946709 32011 tablet_bootstrap.cc:654] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.947149 32012 ts_tablet_manager.cc:1367] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 14:13:30.948311 32012 raft_consensus.cc:348] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:30.948644 32012 raft_consensus.cc:374] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.948753 32010 tablet_bootstrap.cc:654] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.948835 32012 raft_consensus.cc:729] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.949332 32012 consensus_queue.cc:260] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:30.950147 32012 ts_tablet_manager.cc:1398] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:30.950459 32012 tablet_bootstrap.cc:492] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:30.952344 32012 tablet_bootstrap.cc:654] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.954854 32013 tablet_bootstrap.cc:492] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:30.955092 32013 ts_tablet_manager.cc:1367] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20240503 14:13:30.956038 32013 raft_consensus.cc:348] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:30.956316 32010 tablet_bootstrap.cc:492] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:30.956326 32013 raft_consensus.cc:374] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.956393 32011 tablet_bootstrap.cc:492] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:30.956493 32010 ts_tablet_manager.cc:1367] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240503 14:13:30.956533 32013 raft_consensus.cc:729] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.956570 32011 ts_tablet_manager.cc:1367] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240503 14:13:30.957031 32012 tablet_bootstrap.cc:492] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:30.956907 32013 consensus_queue.cc:260] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:30.957262 32012 ts_tablet_manager.cc:1367] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240503 14:13:30.957347 32010 raft_consensus.cc:348] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:30.957576 32010 raft_consensus.cc:374] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.957590 32013 ts_tablet_manager.cc:1398] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:30.957705 32010 raft_consensus.cc:729] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.957872 32013 tablet_bootstrap.cc:492] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:30.958000 32010 consensus_queue.cc:260] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:30.958132 32012 raft_consensus.cc:348] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:30.958381 32012 raft_consensus.cc:374] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.958489 32012 raft_consensus.cc:729] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.958619 32010 ts_tablet_manager.cc:1398] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:30.958778 32012 consensus_queue.cc:260] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:30.959511 32012 ts_tablet_manager.cc:1398] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:30.959851 32012 tablet_bootstrap.cc:492] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:30.960147 32013 tablet_bootstrap.cc:654] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.961100 32011 raft_consensus.cc:348] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:30.961382 32011 raft_consensus.cc:374] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.961416 32012 tablet_bootstrap.cc:654] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.961535 32011 raft_consensus.cc:729] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.961884 32011 consensus_queue.cc:260] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:30.963284 32011 ts_tablet_manager.cc:1398] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:13:30.963743 32011 tablet_bootstrap.cc:492] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:30.965939 32011 tablet_bootstrap.cc:654] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.970048 32011 tablet_bootstrap.cc:492] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:30.970212 32011 ts_tablet_manager.cc:1367] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:13:30.970474 32013 tablet_bootstrap.cc:492] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:30.970999 32011 raft_consensus.cc:348] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:30.971235 32012 tablet_bootstrap.cc:492] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:30.971272 32011 raft_consensus.cc:374] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.971330 32013 ts_tablet_manager.cc:1367] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.001s
I20240503 14:13:30.971418 32012 ts_tablet_manager.cc:1367] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240503 14:13:30.971398 32011 raft_consensus.cc:729] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.971709 32011 consensus_queue.cc:260] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:30.972008 32013 raft_consensus.cc:348] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:30.972317 32013 raft_consensus.cc:374] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.972250 32012 raft_consensus.cc:348] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:30.972419 32011 ts_tablet_manager.cc:1398] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:30.972455 32013 raft_consensus.cc:729] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.972529 32012 raft_consensus.cc:374] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.972632 32012 raft_consensus.cc:729] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.972872 32011 tablet_bootstrap.cc:492] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:30.972940 32012 consensus_queue.cc:260] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:30.973601 32012 ts_tablet_manager.cc:1398] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:30.974233 32011 tablet_bootstrap.cc:654] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.972798 32013 consensus_queue.cc:260] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:30.975695 32013 ts_tablet_manager.cc:1398] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 14:13:30.976713 32011 tablet_bootstrap.cc:492] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:30.976845 32011 ts_tablet_manager.cc:1367] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:30.977509 32011 raft_consensus.cc:348] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:30.977648 32011 raft_consensus.cc:374] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.977741 32011 raft_consensus.cc:729] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.977942 32011 consensus_queue.cc:260] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:30.978425 32011 ts_tablet_manager.cc:1398] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:31.029479 31937 raft_consensus.cc:482] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:31.029706 31937 raft_consensus.cc:504] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:31.030548 31937 leader_election.cc:288] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:31.031116 31332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb75d7d9499475ca60c633c0801d639" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be" is_pre_election: true
I20240503 14:13:31.031422 31332 raft_consensus.cc:2454] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 0.
I20240503 14:13:31.031903 31137 leader_election.cc:302] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82cfdc130877418dbe9d980d368973be, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:31.032285 31937 raft_consensus.cc:2781] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:31.032385 31937 raft_consensus.cc:482] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:31.032531 31937 raft_consensus.cc:3037] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.034978 31937 raft_consensus.cc:504] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:31.035599 31937 leader_election.cc:288] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 election: Requested vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:31.035974 31332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb75d7d9499475ca60c633c0801d639" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be"
I20240503 14:13:31.036233 31332 raft_consensus.cc:3037] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.038157 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb75d7d9499475ca60c633c0801d639" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5"
I20240503 14:13:31.038122 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb75d7d9499475ca60c633c0801d639" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" is_pre_election: true
I20240503 14:13:31.038616 31072 raft_consensus.cc:2454] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 0.
I20240503 14:13:31.039021 31332 raft_consensus.cc:2454] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 1.
I20240503 14:13:31.039649 31137 leader_election.cc:302] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82cfdc130877418dbe9d980d368973be, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters: 6dc4948ed7c740ab89b9e349d151b0e5
I20240503 14:13:31.040036 31937 raft_consensus.cc:2781] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:31.040284 31937 raft_consensus.cc:686] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Becoming Leader. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Running, Role: LEADER
I20240503 14:13:31.040572 31937 consensus_queue.cc:237] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:31.043751 30800 catalog_manager.cc:5484] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: term changed from 0 to 1, leader changed from <none> to e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67). New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:31.063467 31934 raft_consensus.cc:482] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:31.063629 31934 raft_consensus.cc:504] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:31.064352 31934 leader_election.cc:288] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:31.064767 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b5dd5eb46a45989bf9ffe3144bc189" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" is_pre_election: true
I20240503 14:13:31.064985 31202 raft_consensus.cc:2454] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:31.064942 31332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b5dd5eb46a45989bf9ffe3144bc189" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be" is_pre_election: true
I20240503 14:13:31.065207 31332 raft_consensus.cc:2454] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:31.065397 31006 leader_election.cc:302] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:31.065650 31934 raft_consensus.cc:2781] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:31.065743 31934 raft_consensus.cc:482] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:31.065804 31934 raft_consensus.cc:3037] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.067674 31934 raft_consensus.cc:504] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:31.068360 31934 leader_election.cc:288] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Requested vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:31.068698 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b5dd5eb46a45989bf9ffe3144bc189" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce"
I20240503 14:13:31.068882 31202 raft_consensus.cc:3037] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.068806 31332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b5dd5eb46a45989bf9ffe3144bc189" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be"
I20240503 14:13:31.069005 31332 raft_consensus.cc:3037] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.070756 31202 raft_consensus.cc:2454] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:31.070796 31332 raft_consensus.cc:2454] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:31.071146 31006 leader_election.cc:302] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:31.071405 31934 raft_consensus.cc:2781] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:31.071535 31934 raft_consensus.cc:686] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Becoming Leader. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Running, Role: LEADER
I20240503 14:13:31.071797 31934 consensus_queue.cc:237] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:31.074216 30800 catalog_manager.cc:5484] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:31.257756 31934 raft_consensus.cc:482] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:31.257953 31934 raft_consensus.cc:504] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:31.258579 31934 leader_election.cc:288] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:31.259089 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd6dc710e144e61a5fae3911f25ee5c" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" is_pre_election: true
I20240503 14:13:31.259171 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd6dc710e144e61a5fae3911f25ee5c" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:31.259321 31202 raft_consensus.cc:2454] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:31.259418 30940 raft_consensus.cc:2454] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:31.259709 31006 leader_election.cc:302] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:31.259969 31934 raft_consensus.cc:2781] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:31.260066 31934 raft_consensus.cc:482] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:31.260205 31934 raft_consensus.cc:3037] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.262076 31934 raft_consensus.cc:504] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:31.262638 31934 leader_election.cc:288] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Requested vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:31.262989 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd6dc710e144e61a5fae3911f25ee5c" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce"
I20240503 14:13:31.263126 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd6dc710e144e61a5fae3911f25ee5c" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:31.263191 31202 raft_consensus.cc:3037] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.263315 30940 raft_consensus.cc:3037] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.265946 31202 raft_consensus.cc:2454] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:31.266108 30940 raft_consensus.cc:2454] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:31.266358 31006 leader_election.cc:302] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:31.266646 31934 raft_consensus.cc:2781] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:31.266815 31934 raft_consensus.cc:686] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Becoming Leader. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Running, Role: LEADER
I20240503 14:13:31.267079 31934 consensus_queue.cc:237] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:31.269739 30800 catalog_manager.cc:5484] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:31.421146 31937 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.426980 31071 raft_consensus.cc:3037] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.432128 31937 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:31.654537 32027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:31.654856 32027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:31.662000 32027 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:31.675992 31955 consensus_queue.cc:1035] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.677776 31934 consensus_queue.cc:1035] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.680706 31934 consensus_queue.cc:1035] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.690546 31955 consensus_queue.cc:1035] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 201.612 ms
time per row: 0.100806 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:13:32.959080 30796 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36072:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:32.968230 30921 tablet_service.cc:1467] Processing CreateTablet for tablet f6aeaa3fe1cd4127a6144d85b0434057 (DEFAULT_TABLE table=rebalance_test_table_2 [id=026356e4d1494d1c8649c4b1d401565d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.968405 31312 tablet_service.cc:1467] Processing CreateTablet for tablet f6aeaa3fe1cd4127a6144d85b0434057 (DEFAULT_TABLE table=rebalance_test_table_2 [id=026356e4d1494d1c8649c4b1d401565d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.968761 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6aeaa3fe1cd4127a6144d85b0434057. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.968835 31312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6aeaa3fe1cd4127a6144d85b0434057. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.970088 31311 tablet_service.cc:1467] Processing CreateTablet for tablet 5051c4e81ea14146a9f1d82b7f5b5f1d (DEFAULT_TABLE table=rebalance_test_table_2 [id=026356e4d1494d1c8649c4b1d401565d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.970253 31182 tablet_service.cc:1467] Processing CreateTablet for tablet 5051c4e81ea14146a9f1d82b7f5b5f1d (DEFAULT_TABLE table=rebalance_test_table_2 [id=026356e4d1494d1c8649c4b1d401565d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.970466 31311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5051c4e81ea14146a9f1d82b7f5b5f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.970645 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5051c4e81ea14146a9f1d82b7f5b5f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.972643 31310 tablet_service.cc:1467] Processing CreateTablet for tablet 626fcb185af0446b89ae82066e08981d (DEFAULT_TABLE table=rebalance_test_table_2 [id=026356e4d1494d1c8649c4b1d401565d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.972996 31310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 626fcb185af0446b89ae82066e08981d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.973177 31052 tablet_service.cc:1467] Processing CreateTablet for tablet f6aeaa3fe1cd4127a6144d85b0434057 (DEFAULT_TABLE table=rebalance_test_table_2 [id=026356e4d1494d1c8649c4b1d401565d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.973603 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6aeaa3fe1cd4127a6144d85b0434057. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.976928 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 626fcb185af0446b89ae82066e08981d (DEFAULT_TABLE table=rebalance_test_table_2 [id=026356e4d1494d1c8649c4b1d401565d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.977326 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 626fcb185af0446b89ae82066e08981d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.976928 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 5051c4e81ea14146a9f1d82b7f5b5f1d (DEFAULT_TABLE table=rebalance_test_table_2 [id=026356e4d1494d1c8649c4b1d401565d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.980774 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5051c4e81ea14146a9f1d82b7f5b5f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.973201 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 626fcb185af0446b89ae82066e08981d (DEFAULT_TABLE table=rebalance_test_table_2 [id=026356e4d1494d1c8649c4b1d401565d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.984311 32092 tablet_bootstrap.cc:492] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:32.984578 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 626fcb185af0446b89ae82066e08981d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.986577 32092 tablet_bootstrap.cc:654] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.995039 32092 tablet_bootstrap.cc:492] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:32.995342 32092 ts_tablet_manager.cc:1367] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.011s
I20240503 14:13:32.996563 32092 raft_consensus.cc:348] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:32.997025 32092 raft_consensus.cc:374] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.997210 32092 raft_consensus.cc:729] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.997625 32092 consensus_queue.cc:260] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:32.999241 32093 tablet_bootstrap.cc:492] T 5051c4e81ea14146a9f1d82b7f5b5f1d P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:33.001125 32092 ts_tablet_manager.cc:1398] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20240503 14:13:33.001441 32092 tablet_bootstrap.cc:492] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:33.002450 32093 tablet_bootstrap.cc:654] T 5051c4e81ea14146a9f1d82b7f5b5f1d P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.003038 32092 tablet_bootstrap.cc:654] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.007382 32094 tablet_bootstrap.cc:492] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:33.008682 32093 tablet_bootstrap.cc:492] T 5051c4e81ea14146a9f1d82b7f5b5f1d P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:33.008832 32097 tablet_bootstrap.cc:492] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:33.008896 32093 ts_tablet_manager.cc:1367] T 5051c4e81ea14146a9f1d82b7f5b5f1d P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 14:13:33.009805 32094 tablet_bootstrap.cc:654] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.009729 32093 raft_consensus.cc:348] T 5051c4e81ea14146a9f1d82b7f5b5f1d P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:33.010071 32093 raft_consensus.cc:374] T 5051c4e81ea14146a9f1d82b7f5b5f1d P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.010233 32093 raft_consensus.cc:729] T 5051c4e81ea14146a9f1d82b7f5b5f1d P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.010517 32097 tablet_bootstrap.cc:654] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.010555 32093 consensus_queue.cc:260] T 5051c4e81ea14146a9f1d82b7f5b5f1d P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:33.011251 32093 ts_tablet_manager.cc:1398] T 5051c4e81ea14146a9f1d82b7f5b5f1d P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:33.013326 32092 tablet_bootstrap.cc:492] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:33.013500 32092 ts_tablet_manager.cc:1367] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20240503 14:13:33.014181 32097 tablet_bootstrap.cc:492] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:33.014317 32097 ts_tablet_manager.cc:1367] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240503 14:13:33.014292 32092 raft_consensus.cc:348] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:33.014499 32092 raft_consensus.cc:374] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.014575 32092 raft_consensus.cc:729] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.014765 32094 tablet_bootstrap.cc:492] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:33.014899 32094 ts_tablet_manager.cc:1367] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 14:13:33.014832 32092 consensus_queue.cc:260] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:33.015368 32092 ts_tablet_manager.cc:1398] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:33.015628 32094 raft_consensus.cc:348] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:33.015836 32094 raft_consensus.cc:374] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.015934 32094 raft_consensus.cc:729] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.015964 32092 tablet_bootstrap.cc:492] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:33.016184 32094 consensus_queue.cc:260] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:33.018095 32092 tablet_bootstrap.cc:654] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.018303 32094 ts_tablet_manager.cc:1398] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:33.018628 32094 tablet_bootstrap.cc:492] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:33.020058 32097 raft_consensus.cc:348] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:33.020433 32094 tablet_bootstrap.cc:654] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.020617 32097 raft_consensus.cc:374] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.020777 32097 raft_consensus.cc:729] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.021088 32097 consensus_queue.cc:260] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:33.023449 32097 ts_tablet_manager.cc:1398] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20240503 14:13:33.024089 32097 tablet_bootstrap.cc:492] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:33.025776 32097 tablet_bootstrap.cc:654] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.027338 32097 tablet_bootstrap.cc:492] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:33.027495 32097 ts_tablet_manager.cc:1367] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:33.027845 32092 tablet_bootstrap.cc:492] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:33.028118 32092 ts_tablet_manager.cc:1367] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20240503 14:13:33.028343 32094 tablet_bootstrap.cc:492] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:33.028340 32097 raft_consensus.cc:348] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:33.028520 32094 ts_tablet_manager.cc:1367] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20240503 14:13:33.028595 32097 raft_consensus.cc:374] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.028681 32097 raft_consensus.cc:729] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.028921 32097 consensus_queue.cc:260] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:33.029023 32092 raft_consensus.cc:348] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:33.029297 32092 raft_consensus.cc:374] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.029186 32094 raft_consensus.cc:348] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:33.029399 32094 raft_consensus.cc:374] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.029495 32097 ts_tablet_manager.cc:1398] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:33.029491 32094 raft_consensus.cc:729] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.029451 32092 raft_consensus.cc:729] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.029736 32094 consensus_queue.cc:260] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:33.029796 32092 consensus_queue.cc:260] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:33.030269 32094 ts_tablet_manager.cc:1398] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:33.030305 32092 ts_tablet_manager.cc:1398] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:33.030602 32094 tablet_bootstrap.cc:492] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:33.031991 32094 tablet_bootstrap.cc:654] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.035377 32094 tablet_bootstrap.cc:492] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:33.035498 32094 ts_tablet_manager.cc:1367] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:13:33.036083 32094 raft_consensus.cc:348] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:33.036283 32094 raft_consensus.cc:374] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.036381 32094 raft_consensus.cc:729] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.036597 32094 consensus_queue.cc:260] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:33.037043 32094 ts_tablet_manager.cc:1398] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:13:33.144459 32096 raft_consensus.cc:482] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.144685 32096 raft_consensus.cc:504] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:33.146220 32096 leader_election.cc:288] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:33.160360 32096 raft_consensus.cc:482] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.160586 32096 raft_consensus.cc:504] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:33.161422 32096 leader_election.cc:288] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:33.167757 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6aeaa3fe1cd4127a6144d85b0434057" candidate_uuid: "0dc902d91ccc435a82a5ede4a7729811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" is_pre_election: true
I20240503 14:13:33.168048 31071 raft_consensus.cc:2454] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dc902d91ccc435a82a5ede4a7729811 in term 0.
I20240503 14:13:33.169005 30877 leader_election.cc:302] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 6dc4948ed7c740ab89b9e349d151b0e5; no voters:
I20240503 14:13:33.169515 32096 raft_consensus.cc:2781] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.169617 32096 raft_consensus.cc:482] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.169503 31332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6aeaa3fe1cd4127a6144d85b0434057" candidate_uuid: "0dc902d91ccc435a82a5ede4a7729811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be" is_pre_election: true
I20240503 14:13:33.169736 32096 raft_consensus.cc:3037] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.169505 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5051c4e81ea14146a9f1d82b7f5b5f1d" candidate_uuid: "0dc902d91ccc435a82a5ede4a7729811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be" is_pre_election: true
I20240503 14:13:33.169799 31332 raft_consensus.cc:2454] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dc902d91ccc435a82a5ede4a7729811 in term 0.
I20240503 14:13:33.169958 31331 raft_consensus.cc:2454] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dc902d91ccc435a82a5ede4a7729811 in term 0.
I20240503 14:13:33.170687 30876 leader_election.cc:302] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 82cfdc130877418dbe9d980d368973be; no voters:
I20240503 14:13:33.171511 32113 raft_consensus.cc:2781] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.171643 32113 raft_consensus.cc:482] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.171768 32113 raft_consensus.cc:3037] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.173018 32096 raft_consensus.cc:504] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:33.174053 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6aeaa3fe1cd4127a6144d85b0434057" candidate_uuid: "0dc902d91ccc435a82a5ede4a7729811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be"
I20240503 14:13:33.174185 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6aeaa3fe1cd4127a6144d85b0434057" candidate_uuid: "0dc902d91ccc435a82a5ede4a7729811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5"
I20240503 14:13:33.174279 31331 raft_consensus.cc:3037] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.174389 31071 raft_consensus.cc:3037] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.174613 32113 raft_consensus.cc:504] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:33.175248 32113 leader_election.cc:288] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [CANDIDATE]: Term 1 election: Requested vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:33.175487 32096 leader_election.cc:288] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [CANDIDATE]: Term 1 election: Requested vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:33.175786 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5051c4e81ea14146a9f1d82b7f5b5f1d" candidate_uuid: "0dc902d91ccc435a82a5ede4a7729811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce"
I20240503 14:13:33.175982 31201 raft_consensus.cc:3037] T 5051c4e81ea14146a9f1d82b7f5b5f1d P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.176393 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5051c4e81ea14146a9f1d82b7f5b5f1d" candidate_uuid: "0dc902d91ccc435a82a5ede4a7729811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" is_pre_election: true
I20240503 14:13:33.176512 31331 raft_consensus.cc:2454] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dc902d91ccc435a82a5ede4a7729811 in term 1.
I20240503 14:13:33.177258 31071 raft_consensus.cc:2454] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dc902d91ccc435a82a5ede4a7729811 in term 1.
I20240503 14:13:33.177448 30876 leader_election.cc:302] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 82cfdc130877418dbe9d980d368973be; no voters:
I20240503 14:13:33.177670 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5051c4e81ea14146a9f1d82b7f5b5f1d" candidate_uuid: "0dc902d91ccc435a82a5ede4a7729811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be"
I20240503 14:13:33.177850 31331 raft_consensus.cc:3037] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.178572 31201 raft_consensus.cc:2454] T 5051c4e81ea14146a9f1d82b7f5b5f1d P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dc902d91ccc435a82a5ede4a7729811 in term 1.
I20240503 14:13:33.178822 32096 raft_consensus.cc:2781] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.179021 30875 leader_election.cc:302] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:33.179060 32096 raft_consensus.cc:686] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 LEADER]: Becoming Leader. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Running, Role: LEADER
I20240503 14:13:33.179292 32113 raft_consensus.cc:2781] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.179605 32096 consensus_queue.cc:237] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:33.179875 32113 raft_consensus.cc:686] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [term 1 LEADER]: Becoming Leader. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Running, Role: LEADER
I20240503 14:13:33.180471 31331 raft_consensus.cc:2454] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dc902d91ccc435a82a5ede4a7729811 in term 1.
I20240503 14:13:33.180457 32113 consensus_queue.cc:237] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:33.184312 30796 catalog_manager.cc:5484] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "0dc902d91ccc435a82a5ede4a7729811" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:33.188680 30796 catalog_manager.cc:5484] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "0dc902d91ccc435a82a5ede4a7729811" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:33.211086 32102 raft_consensus.cc:482] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.211305 32102 raft_consensus.cc:504] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:33.212962 32102 leader_election.cc:288] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:33.222087 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626fcb185af0446b89ae82066e08981d" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" is_pre_election: true
I20240503 14:13:33.222214 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626fcb185af0446b89ae82066e08981d" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:33.222340 31071 raft_consensus.cc:2454] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 0.
I20240503 14:13:33.222477 30940 raft_consensus.cc:2454] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 0.
I20240503 14:13:33.222868 31268 leader_election.cc:302] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, 82cfdc130877418dbe9d980d368973be; no voters:
I20240503 14:13:33.223269 32102 raft_consensus.cc:2781] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.223404 32102 raft_consensus.cc:482] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.223508 32102 raft_consensus.cc:3037] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.225476 32102 raft_consensus.cc:504] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:33.226127 32102 leader_election.cc:288] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 election: Requested vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:33.226464 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626fcb185af0446b89ae82066e08981d" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:33.226596 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626fcb185af0446b89ae82066e08981d" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5"
I20240503 14:13:33.226694 30940 raft_consensus.cc:3037] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.226817 31071 raft_consensus.cc:3037] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.228674 30940 raft_consensus.cc:2454] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 1.
I20240503 14:13:33.228680 31071 raft_consensus.cc:2454] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 1.
I20240503 14:13:33.229053 31268 leader_election.cc:302] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, 82cfdc130877418dbe9d980d368973be; no voters:
I20240503 14:13:33.229363 32102 raft_consensus.cc:2781] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.229764 32102 raft_consensus.cc:686] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Becoming Leader. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Running, Role: LEADER
I20240503 14:13:33.230238 32102 consensus_queue.cc:237] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:33.233574 30796 catalog_manager.cc:5484] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be reported cstate change: term changed from 0 to 1, leader changed from <none> to 82cfdc130877418dbe9d980d368973be (127.22.242.68). New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:33.561048 32096 consensus_queue.cc:1035] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.566920 32114 consensus_queue.cc:1035] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:33.630353 32122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:33.630667 32122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:33.638083 32122 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 14:13:33.684283 32114 consensus_queue.cc:1035] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.689018 32096 consensus_queue.cc:1035] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.721647 32102 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.750253 32102 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 250.173 ms
time per row: 0.125086 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:13:34.968334 30798 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36072:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:34.976855 31182 tablet_service.cc:1467] Processing CreateTablet for tablet f3330e28c2d7434699b78070dd92ed73 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ca6b9f7a0ded4c3aa1ea18aa2055c41b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:34.977067 30920 tablet_service.cc:1467] Processing CreateTablet for tablet f3330e28c2d7434699b78070dd92ed73 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ca6b9f7a0ded4c3aa1ea18aa2055c41b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:34.977329 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3330e28c2d7434699b78070dd92ed73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.977516 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3330e28c2d7434699b78070dd92ed73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.978623 31310 tablet_service.cc:1467] Processing CreateTablet for tablet f3e80559017d4a6ba16d34e0b7aab085 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ca6b9f7a0ded4c3aa1ea18aa2055c41b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:34.979017 31310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e80559017d4a6ba16d34e0b7aab085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.982417 32188 tablet_bootstrap.cc:492] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:34.982677 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 91a7b8a53140429b956178d43478a541 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ca6b9f7a0ded4c3aa1ea18aa2055c41b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:34.983031 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a7b8a53140429b956178d43478a541. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.984203 32188 tablet_bootstrap.cc:654] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.984436 31312 tablet_service.cc:1467] Processing CreateTablet for tablet 91a7b8a53140429b956178d43478a541 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ca6b9f7a0ded4c3aa1ea18aa2055c41b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:34.984946 31312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a7b8a53140429b956178d43478a541. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.985227 31180 tablet_service.cc:1467] Processing CreateTablet for tablet f3e80559017d4a6ba16d34e0b7aab085 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ca6b9f7a0ded4c3aa1ea18aa2055c41b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:34.985577 31180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e80559017d4a6ba16d34e0b7aab085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.985738 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 91a7b8a53140429b956178d43478a541 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ca6b9f7a0ded4c3aa1ea18aa2055c41b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:34.986048 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a7b8a53140429b956178d43478a541. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.988787 31051 tablet_service.cc:1467] Processing CreateTablet for tablet f3330e28c2d7434699b78070dd92ed73 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ca6b9f7a0ded4c3aa1ea18aa2055c41b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:34.989300 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3330e28c2d7434699b78070dd92ed73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.991411 31052 tablet_service.cc:1467] Processing CreateTablet for tablet f3e80559017d4a6ba16d34e0b7aab085 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ca6b9f7a0ded4c3aa1ea18aa2055c41b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:34.991858 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e80559017d4a6ba16d34e0b7aab085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.992439 32189 tablet_bootstrap.cc:492] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:34.993907 32189 tablet_bootstrap.cc:654] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.002779 32189 tablet_bootstrap.cc:492] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:35.002986 32189 ts_tablet_manager.cc:1367] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240503 14:13:35.005515 32189 raft_consensus.cc:348] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.005985 32189 raft_consensus.cc:374] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.006162 32189 raft_consensus.cc:729] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.006561 32189 consensus_queue.cc:260] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.010722 32193 tablet_bootstrap.cc:492] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:35.012512 32192 tablet_bootstrap.cc:492] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:35.012540 32189 ts_tablet_manager.cc:1398] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 14:13:35.013024 32193 tablet_bootstrap.cc:654] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.013213 32188 tablet_bootstrap.cc:492] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:35.013455 32188 ts_tablet_manager.cc:1367] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.016s
I20240503 14:13:35.014139 32192 tablet_bootstrap.cc:654] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.014369 32188 raft_consensus.cc:348] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.014655 32188 raft_consensus.cc:374] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.014801 32188 raft_consensus.cc:729] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.015139 32188 consensus_queue.cc:260] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.015656 32189 tablet_bootstrap.cc:492] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:35.016789 32188 ts_tablet_manager.cc:1398] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:35.017236 32188 tablet_bootstrap.cc:492] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:35.017966 32189 tablet_bootstrap.cc:654] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.019421 32188 tablet_bootstrap.cc:654] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.020351 32189 tablet_bootstrap.cc:492] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:35.020438 32192 tablet_bootstrap.cc:492] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:35.020555 32189 ts_tablet_manager.cc:1367] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 14:13:35.020591 32192 ts_tablet_manager.cc:1367] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20240503 14:13:35.021395 32189 raft_consensus.cc:348] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:35.021400 32192 raft_consensus.cc:348] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.021618 32192 raft_consensus.cc:374] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.021623 32189 raft_consensus.cc:374] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.021701 32189 raft_consensus.cc:729] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.021709 32192 raft_consensus.cc:729] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.021996 32189 consensus_queue.cc:260] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:35.022326 32193 tablet_bootstrap.cc:492] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:35.022478 32193 ts_tablet_manager.cc:1367] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.003s
I20240503 14:13:35.022598 32189 ts_tablet_manager.cc:1398] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:35.023278 32193 raft_consensus.cc:348] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.021975 32192 consensus_queue.cc:260] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.023563 32193 raft_consensus.cc:374] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.023655 32193 raft_consensus.cc:729] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.023910 32192 ts_tablet_manager.cc:1398] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:35.023893 32193 consensus_queue.cc:260] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.024195 32192 tablet_bootstrap.cc:492] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:35.024533 32193 ts_tablet_manager.cc:1398] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:35.024933 32193 tablet_bootstrap.cc:492] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:35.028724 32192 tablet_bootstrap.cc:654] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.030886 32192 tablet_bootstrap.cc:492] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:35.031062 32192 ts_tablet_manager.cc:1367] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:13:35.031782 32192 raft_consensus.cc:348] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.032001 32192 raft_consensus.cc:374] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.032071 32193 tablet_bootstrap.cc:654] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.032083 32192 raft_consensus.cc:729] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.032334 32192 consensus_queue.cc:260] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.032845 32192 ts_tablet_manager.cc:1398] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:35.034102 32188 tablet_bootstrap.cc:492] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:35.034302 32188 ts_tablet_manager.cc:1367] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.007s
I20240503 14:13:35.034477 32193 tablet_bootstrap.cc:492] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:35.034636 32193 ts_tablet_manager.cc:1367] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20240503 14:13:35.035108 32188 raft_consensus.cc:348] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.035362 32188 raft_consensus.cc:374] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.035349 32193 raft_consensus.cc:348] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:35.035482 32188 raft_consensus.cc:729] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.035581 32193 raft_consensus.cc:374] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.035682 32193 raft_consensus.cc:729] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.035766 32188 consensus_queue.cc:260] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.036366 32188 ts_tablet_manager.cc:1398] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 14:13:35.036695 32188 tablet_bootstrap.cc:492] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:35.036742 32193 consensus_queue.cc:260] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:35.037320 32193 ts_tablet_manager.cc:1398] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:35.038245 32188 tablet_bootstrap.cc:654] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.040710 32188 tablet_bootstrap.cc:492] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:35.040843 32188 ts_tablet_manager.cc:1367] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:35.041716 32188 raft_consensus.cc:348] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:35.041951 32188 raft_consensus.cc:374] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.042078 32188 raft_consensus.cc:729] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.042295 32188 consensus_queue.cc:260] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:35.042759 32188 ts_tablet_manager.cc:1398] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:35.098963 31949 raft_consensus.cc:482] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:35.099164 31949 raft_consensus.cc:504] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.099843 31949 leader_election.cc:288] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:35.100514 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3330e28c2d7434699b78070dd92ed73" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" is_pre_election: true
I20240503 14:13:35.100514 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3330e28c2d7434699b78070dd92ed73" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:35.100776 30940 raft_consensus.cc:2454] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 0.
I20240503 14:13:35.100776 31072 raft_consensus.cc:2454] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 0.
I20240503 14:13:35.101264 31135 leader_election.cc:302] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:35.101563 31949 raft_consensus.cc:2781] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:35.101683 31949 raft_consensus.cc:482] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:35.101775 31949 raft_consensus.cc:3037] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.103693 31949 raft_consensus.cc:504] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.104308 31949 leader_election.cc:288] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 election: Requested vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:35.104727 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3330e28c2d7434699b78070dd92ed73" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5"
I20240503 14:13:35.104727 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3330e28c2d7434699b78070dd92ed73" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:35.104925 30940 raft_consensus.cc:3037] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.104925 31072 raft_consensus.cc:3037] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.107719 30940 raft_consensus.cc:2454] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 1.
I20240503 14:13:35.107719 31072 raft_consensus.cc:2454] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 1.
I20240503 14:13:35.108204 31138 leader_election.cc:302] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:35.108502 31949 raft_consensus.cc:2781] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:35.108923 31949 raft_consensus.cc:686] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Becoming Leader. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Running, Role: LEADER
I20240503 14:13:35.109259 31949 consensus_queue.cc:237] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.112052 30798 catalog_manager.cc:5484] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: term changed from 0 to 1, leader changed from <none> to e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67). New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:35.158329 31949 raft_consensus.cc:482] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:35.158607 31949 raft_consensus.cc:504] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:35.159298 31949 leader_election.cc:288] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:35.159792 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7b8a53140429b956178d43478a541" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:35.159847 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7b8a53140429b956178d43478a541" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be" is_pre_election: true
I20240503 14:13:35.160040 30940 raft_consensus.cc:2454] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 0.
I20240503 14:13:35.160079 31331 raft_consensus.cc:2454] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 0.
I20240503 14:13:35.160434 31135 leader_election.cc:302] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:35.160719 31949 raft_consensus.cc:2781] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:35.160797 31949 raft_consensus.cc:482] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:35.160900 31949 raft_consensus.cc:3037] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.162755 31949 raft_consensus.cc:504] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:35.163302 31949 leader_election.cc:288] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 election: Requested vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:35.163692 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7b8a53140429b956178d43478a541" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:35.163712 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7b8a53140429b956178d43478a541" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be"
I20240503 14:13:35.163903 30940 raft_consensus.cc:3037] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.163914 31331 raft_consensus.cc:3037] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.165720 30940 raft_consensus.cc:2454] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 1.
I20240503 14:13:35.165803 31331 raft_consensus.cc:2454] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 1.
I20240503 14:13:35.166221 31137 leader_election.cc:302] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82cfdc130877418dbe9d980d368973be, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:35.166486 31949 raft_consensus.cc:2781] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:35.166657 31949 raft_consensus.cc:686] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Becoming Leader. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Running, Role: LEADER
I20240503 14:13:35.166945 31949 consensus_queue.cc:237] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:35.169729 30798 catalog_manager.cc:5484] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: term changed from 0 to 1, leader changed from <none> to e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67). New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:35.249524 32042 raft_consensus.cc:482] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:35.249751 32042 raft_consensus.cc:504] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.250509 32042 leader_election.cc:288] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:35.250916 31949 raft_consensus.cc:482] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:35.251052 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e80559017d4a6ba16d34e0b7aab085" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" is_pre_election: true
I20240503 14:13:35.251118 31949 raft_consensus.cc:504] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.251447 31201 raft_consensus.cc:2454] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:35.251787 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e80559017d4a6ba16d34e0b7aab085" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be" is_pre_election: true
I20240503 14:13:35.252005 31331 raft_consensus.cc:2454] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:35.252279 31006 leader_election.cc:302] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:35.252265 31332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e80559017d4a6ba16d34e0b7aab085" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be" is_pre_election: true
I20240503 14:13:35.252475 31332 raft_consensus.cc:2454] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 0.
I20240503 14:13:35.252674 32042 raft_consensus.cc:2781] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:35.252779 32042 raft_consensus.cc:482] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:35.252856 32042 raft_consensus.cc:3037] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.252866 31137 leader_election.cc:302] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82cfdc130877418dbe9d980d368973be, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:35.253258 32201 raft_consensus.cc:2781] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:35.253357 32201 raft_consensus.cc:482] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:35.253473 32201 raft_consensus.cc:3037] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.253641 31949 leader_election.cc:288] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:35.254192 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e80559017d4a6ba16d34e0b7aab085" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" is_pre_election: true
I20240503 14:13:35.255597 32042 raft_consensus.cc:504] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.255975 31072 raft_consensus.cc:2379] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in current term 1: Already voted for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in this term.
I20240503 14:13:35.256080 32201 raft_consensus.cc:504] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.256628 31332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e80559017d4a6ba16d34e0b7aab085" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be"
I20240503 14:13:35.256824 31332 raft_consensus.cc:3037] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.256855 32201 leader_election.cc:288] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 election: Requested vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:35.256889 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e80559017d4a6ba16d34e0b7aab085" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce"
I20240503 14:13:35.257081 32042 leader_election.cc:288] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Requested vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:35.257169 31201 raft_consensus.cc:2379] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6dc4948ed7c740ab89b9e349d151b0e5 in current term 1: Already voted for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in this term.
I20240503 14:13:35.257205 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e80559017d4a6ba16d34e0b7aab085" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be"
I20240503 14:13:35.258074 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e80559017d4a6ba16d34e0b7aab085" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5"
I20240503 14:13:35.258329 31072 raft_consensus.cc:2379] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in current term 1: Already voted for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in this term.
I20240503 14:13:35.258697 31138 leader_election.cc:302] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters: 6dc4948ed7c740ab89b9e349d151b0e5, 82cfdc130877418dbe9d980d368973be
I20240503 14:13:35.258961 32201 raft_consensus.cc:2726] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:13:35.259126 31332 raft_consensus.cc:2454] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:35.259460 31007 leader_election.cc:302] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, 82cfdc130877418dbe9d980d368973be; no voters: e7badbe63a8a4d9ca0c78818c6f4c8ce
I20240503 14:13:35.259711 32042 raft_consensus.cc:2781] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:35.260025 32042 raft_consensus.cc:686] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Becoming Leader. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Running, Role: LEADER
I20240503 14:13:35.260346 32042 consensus_queue.cc:237] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:35.262854 30798 catalog_manager.cc:5484] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:35.553537 32201 consensus_queue.cc:1035] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.566820 32201 consensus_queue.cc:1035] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.610610 31949 consensus_queue.cc:1035] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.616750 31949 consensus_queue.cc:1035] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:35.655396 32206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:35.655725 32206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:35.663277 32206 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 14:13:35.803535 31332 raft_consensus.cc:1261] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:35.804661 32042 consensus_queue.cc:1035] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.828305 31201 raft_consensus.cc:1261] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:35.840852 32253 consensus_queue.cc:1035] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.941540 32248 mvcc.cc:205] Tried to move back new op lower bound from 7023598042322759680 to 7023598040108441600. Current Snapshot: MvccSnapshot[applied={T|T < 7023598042322759680}]
INSERT report
rows total: 2000
time total: 243.645 ms
time per row: 0.121822 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:13:37.008483 30798 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36072:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:37.017519 31051 tablet_service.cc:1467] Processing CreateTablet for tablet dd5869fd8f8d41d99cf531edb7df3750 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f05910cae90c4382a503986593cff68f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.017665 30919 tablet_service.cc:1467] Processing CreateTablet for tablet dd5869fd8f8d41d99cf531edb7df3750 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f05910cae90c4382a503986593cff68f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.018010 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5869fd8f8d41d99cf531edb7df3750. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.018100 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5869fd8f8d41d99cf531edb7df3750. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.018384 31182 tablet_service.cc:1467] Processing CreateTablet for tablet dd5869fd8f8d41d99cf531edb7df3750 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f05910cae90c4382a503986593cff68f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.018790 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5869fd8f8d41d99cf531edb7df3750. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.019465 31310 tablet_service.cc:1467] Processing CreateTablet for tablet da9dd5cd81114817b0f898c6b8fb3c07 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f05910cae90c4382a503986593cff68f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.019855 31310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9dd5cd81114817b0f898c6b8fb3c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.020561 31312 tablet_service.cc:1467] Processing CreateTablet for tablet 23476c4bcf6e4b4085d59d29739b6edb (DEFAULT_TABLE table=rebalance_test_table_4 [id=f05910cae90c4382a503986593cff68f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.020879 31312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23476c4bcf6e4b4085d59d29739b6edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.025002 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 23476c4bcf6e4b4085d59d29739b6edb (DEFAULT_TABLE table=rebalance_test_table_4 [id=f05910cae90c4382a503986593cff68f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.025204 30920 tablet_service.cc:1467] Processing CreateTablet for tablet da9dd5cd81114817b0f898c6b8fb3c07 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f05910cae90c4382a503986593cff68f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.025399 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23476c4bcf6e4b4085d59d29739b6edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.025547 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9dd5cd81114817b0f898c6b8fb3c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.028597 31052 tablet_service.cc:1467] Processing CreateTablet for tablet da9dd5cd81114817b0f898c6b8fb3c07 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f05910cae90c4382a503986593cff68f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.028975 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9dd5cd81114817b0f898c6b8fb3c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.032533 31050 tablet_service.cc:1467] Processing CreateTablet for tablet 23476c4bcf6e4b4085d59d29739b6edb (DEFAULT_TABLE table=rebalance_test_table_4 [id=f05910cae90c4382a503986593cff68f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.032903 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23476c4bcf6e4b4085d59d29739b6edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.040354 32275 tablet_bootstrap.cc:492] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:37.041875 32276 tablet_bootstrap.cc:492] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:37.043033 32275 tablet_bootstrap.cc:654] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.043715 32274 tablet_bootstrap.cc:492] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:37.045143 32276 tablet_bootstrap.cc:654] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.046643 32274 tablet_bootstrap.cc:654] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.048725 32277 tablet_bootstrap.cc:492] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:37.051054 32277 tablet_bootstrap.cc:654] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.053346 32274 tablet_bootstrap.cc:492] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:37.053346 32275 tablet_bootstrap.cc:492] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:37.053496 32275 ts_tablet_manager.cc:1367] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240503 14:13:37.053565 32274 ts_tablet_manager.cc:1367] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240503 14:13:37.054348 32275 raft_consensus.cc:348] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:37.054348 32274 raft_consensus.cc:348] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:37.054555 32274 raft_consensus.cc:374] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.054661 32274 raft_consensus.cc:729] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.054689 32275 raft_consensus.cc:374] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.054829 32275 raft_consensus.cc:729] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.054955 32274 consensus_queue.cc:260] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:37.055193 32275 consensus_queue.cc:260] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:37.055580 32274 ts_tablet_manager.cc:1398] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:37.056002 32274 tablet_bootstrap.cc:492] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:37.056643 32275 ts_tablet_manager.cc:1398] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:13:37.056826 32276 tablet_bootstrap.cc:492] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:37.056984 32276 ts_tablet_manager.cc:1367] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20240503 14:13:37.057031 32275 tablet_bootstrap.cc:492] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:37.057586 32274 tablet_bootstrap.cc:654] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.057788 32276 raft_consensus.cc:348] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:37.058046 32276 raft_consensus.cc:374] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.058176 32276 raft_consensus.cc:729] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.058480 32276 consensus_queue.cc:260] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:37.059224 32276 ts_tablet_manager.cc:1398] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:37.059511 32276 tablet_bootstrap.cc:492] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:37.062417 32276 tablet_bootstrap.cc:654] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.064489 32277 tablet_bootstrap.cc:492] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:37.064756 32277 ts_tablet_manager.cc:1367] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.004s
I20240503 14:13:37.065675 32277 raft_consensus.cc:348] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:37.065946 32277 raft_consensus.cc:374] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.066135 32277 raft_consensus.cc:729] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.066524 32277 consensus_queue.cc:260] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:37.067459 32277 ts_tablet_manager.cc:1398] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:37.068370 32277 tablet_bootstrap.cc:492] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:37.068560 32275 tablet_bootstrap.cc:654] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.070091 32277 tablet_bootstrap.cc:654] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.072410 32276 tablet_bootstrap.cc:492] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:37.072678 32276 ts_tablet_manager.cc:1367] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20240503 14:13:37.073715 32276 raft_consensus.cc:348] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:37.074088 32276 raft_consensus.cc:374] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.074234 32276 raft_consensus.cc:729] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.074630 32276 consensus_queue.cc:260] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:37.075310 32276 ts_tablet_manager.cc:1398] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:37.076588 32274 tablet_bootstrap.cc:492] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:37.076793 32274 ts_tablet_manager.cc:1367] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.013s
I20240503 14:13:37.077759 32274 raft_consensus.cc:348] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:37.078025 32274 raft_consensus.cc:374] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.078115 32274 raft_consensus.cc:729] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.078392 32274 consensus_queue.cc:260] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:37.079530 32274 ts_tablet_manager.cc:1398] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:37.080713 32277 tablet_bootstrap.cc:492] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:37.080832 32275 tablet_bootstrap.cc:492] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:37.080929 32277 ts_tablet_manager.cc:1367] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20240503 14:13:37.081112 32275 ts_tablet_manager.cc:1367] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.005s
I20240503 14:13:37.081750 32277 raft_consensus.cc:348] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:37.081982 32277 raft_consensus.cc:374] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.082069 32277 raft_consensus.cc:729] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.082019 32275 raft_consensus.cc:348] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:37.082288 32275 raft_consensus.cc:374] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.082373 32275 raft_consensus.cc:729] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.082352 32277 consensus_queue.cc:260] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:37.082612 32275 consensus_queue.cc:260] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:37.082906 32277 ts_tablet_manager.cc:1398] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:37.083148 32275 ts_tablet_manager.cc:1398] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:37.083534 32275 tablet_bootstrap.cc:492] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:37.084991 32275 tablet_bootstrap.cc:654] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.086930 32275 tablet_bootstrap.cc:492] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:37.087082 32275 ts_tablet_manager.cc:1367] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:37.087711 32275 raft_consensus.cc:348] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:37.087868 32275 raft_consensus.cc:374] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.087970 32275 raft_consensus.cc:729] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.088234 32275 consensus_queue.cc:260] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:37.088780 32275 ts_tablet_manager.cc:1398] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:37.129422 32253 raft_consensus.cc:482] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:37.129642 32253 raft_consensus.cc:504] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:37.130261 32253 leader_election.cc:288] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:37.130817 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9dd5cd81114817b0f898c6b8fb3c07" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:37.131095 30939 raft_consensus.cc:2454] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:37.131610 31005 leader_election.cc:302] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 6dc4948ed7c740ab89b9e349d151b0e5; no voters:
I20240503 14:13:37.131888 32253 raft_consensus.cc:2781] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:37.131987 32253 raft_consensus.cc:482] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:37.132093 32253 raft_consensus.cc:3037] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.132001 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9dd5cd81114817b0f898c6b8fb3c07" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be" is_pre_election: true
I20240503 14:13:37.132313 31331 raft_consensus.cc:2454] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:37.134914 32253 raft_consensus.cc:504] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:37.136328 32253 leader_election.cc:288] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Requested vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:37.140725 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9dd5cd81114817b0f898c6b8fb3c07" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be"
I20240503 14:13:37.140772 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9dd5cd81114817b0f898c6b8fb3c07" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:37.140976 30939 raft_consensus.cc:3037] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.141000 31331 raft_consensus.cc:3037] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.143726 30939 raft_consensus.cc:2454] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:37.144098 31005 leader_election.cc:302] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 6dc4948ed7c740ab89b9e349d151b0e5; no voters:
I20240503 14:13:37.144311 31331 raft_consensus.cc:2454] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:37.144438 32253 raft_consensus.cc:2781] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:37.144913 32253 raft_consensus.cc:686] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Becoming Leader. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Running, Role: LEADER
I20240503 14:13:37.145218 32253 consensus_queue.cc:237] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:37.148067 30794 catalog_manager.cc:5484] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:37.244211 32102 raft_consensus.cc:482] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:37.244432 32102 raft_consensus.cc:504] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:37.245225 32102 leader_election.cc:288] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:37.246168 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23476c4bcf6e4b4085d59d29739b6edb" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" is_pre_election: true
I20240503 14:13:37.246426 31072 raft_consensus.cc:2454] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 0.
I20240503 14:13:37.246917 31268 leader_election.cc:302] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, 82cfdc130877418dbe9d980d368973be; no voters:
I20240503 14:13:37.247201 32102 raft_consensus.cc:2781] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:37.247295 32102 raft_consensus.cc:482] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:37.247383 32102 raft_consensus.cc:3037] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.250630 32102 raft_consensus.cc:504] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:37.251778 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23476c4bcf6e4b4085d59d29739b6edb" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5"
I20240503 14:13:37.251997 31072 raft_consensus.cc:3037] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.254745 31072 raft_consensus.cc:2454] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 1.
I20240503 14:13:37.255216 31268 leader_election.cc:302] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, 82cfdc130877418dbe9d980d368973be; no voters:
I20240503 14:13:37.256894 32102 leader_election.cc:288] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 election: Requested vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:37.257076 32102 raft_consensus.cc:2781] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:37.257220 32102 raft_consensus.cc:686] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Becoming Leader. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Running, Role: LEADER
I20240503 14:13:37.257508 32102 consensus_queue.cc:237] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:37.260586 30794 catalog_manager.cc:5484] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be reported cstate change: term changed from 0 to 1, leader changed from <none> to 82cfdc130877418dbe9d980d368973be (127.22.242.68). New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:37.264401 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23476c4bcf6e4b4085d59d29739b6edb" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce"
I20240503 14:13:37.264360 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23476c4bcf6e4b4085d59d29739b6edb" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" is_pre_election: true
I20240503 14:13:37.264613 31201 raft_consensus.cc:2454] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 0.
I20240503 14:13:37.277797 31949 raft_consensus.cc:482] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:37.277988 31949 raft_consensus.cc:504] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:37.278589 31949 leader_election.cc:288] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:37.279139 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5869fd8f8d41d99cf531edb7df3750" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:37.279177 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5869fd8f8d41d99cf531edb7df3750" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" is_pre_election: true
I20240503 14:13:37.279397 31072 raft_consensus.cc:2454] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 0.
I20240503 14:13:37.279399 30939 raft_consensus.cc:2454] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 0.
I20240503 14:13:37.279853 31138 leader_election.cc:302] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:37.280138 31949 raft_consensus.cc:2781] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:37.280267 31949 raft_consensus.cc:482] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:37.280378 31949 raft_consensus.cc:3037] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.282229 31949 raft_consensus.cc:504] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:37.282789 31949 leader_election.cc:288] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 election: Requested vote from peers 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:37.283207 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5869fd8f8d41d99cf531edb7df3750" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5"
I20240503 14:13:37.283390 31072 raft_consensus.cc:3037] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.283406 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5869fd8f8d41d99cf531edb7df3750" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:37.283679 30939 raft_consensus.cc:3037] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.285316 31072 raft_consensus.cc:2454] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 1.
I20240503 14:13:37.285801 31138 leader_election.cc:302] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:37.285866 30939 raft_consensus.cc:2454] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 1.
I20240503 14:13:37.286069 31949 raft_consensus.cc:2781] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:37.286300 31949 raft_consensus.cc:686] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Becoming Leader. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Running, Role: LEADER
I20240503 14:13:37.286666 31949 consensus_queue.cc:237] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:37.289563 30794 catalog_manager.cc:5484] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: term changed from 0 to 1, leader changed from <none> to e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67). New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:37.547860 32253 consensus_queue.cc:1035] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.555296 32288 consensus_queue.cc:1035] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.656203 31201 raft_consensus.cc:3037] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.658789 32292 consensus_queue.cc:1035] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.664211 32292 consensus_queue.cc:1035] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:37.677059 32295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:37.677376 32295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:37.685253 32295 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 14:13:37.818984 30939 raft_consensus.cc:1261] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:37.819762 31949 consensus_queue.cc:1035] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.820858 31072 raft_consensus.cc:1261] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:37.822198 32339 consensus_queue.cc:1035] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 312.162 ms
time per row: 0.156081 ms
I20240503 14:13:38.143200 30799 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36072:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:38.152707 31181 tablet_service.cc:1467] Processing CreateTablet for tablet bc77a7da906b419085258f72667ff185 (DEFAULT_TABLE table=rebalance_test_table_5 [id=04f8dfb749c04486895d6b0f7c206446]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.153189 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc77a7da906b419085258f72667ff185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.153218 31182 tablet_service.cc:1467] Processing CreateTablet for tablet 55cf34de74694f47ad3f9f4d9bdffae4 (DEFAULT_TABLE table=rebalance_test_table_5 [id=04f8dfb749c04486895d6b0f7c206446]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.153589 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55cf34de74694f47ad3f9f4d9bdffae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.154484 31050 tablet_service.cc:1467] Processing CreateTablet for tablet d6b9a10c474f46bda0310ce661022b42 (DEFAULT_TABLE table=rebalance_test_table_5 [id=04f8dfb749c04486895d6b0f7c206446]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.154902 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b9a10c474f46bda0310ce661022b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.157699 31312 tablet_service.cc:1467] Processing CreateTablet for tablet 55cf34de74694f47ad3f9f4d9bdffae4 (DEFAULT_TABLE table=rebalance_test_table_5 [id=04f8dfb749c04486895d6b0f7c206446]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.158143 31312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55cf34de74694f47ad3f9f4d9bdffae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.161947 32365 tablet_bootstrap.cc:492] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:38.162671 32364 tablet_bootstrap.cc:492] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:38.163795 32365 tablet_bootstrap.cc:654] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.163751 31310 tablet_service.cc:1467] Processing CreateTablet for tablet bc77a7da906b419085258f72667ff185 (DEFAULT_TABLE table=rebalance_test_table_5 [id=04f8dfb749c04486895d6b0f7c206446]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.164124 31310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc77a7da906b419085258f72667ff185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.164240 32364 tablet_bootstrap.cc:654] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.164857 32366 tablet_bootstrap.cc:492] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:38.167294 32366 tablet_bootstrap.cc:654] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.170646 30919 tablet_service.cc:1467] Processing CreateTablet for tablet bc77a7da906b419085258f72667ff185 (DEFAULT_TABLE table=rebalance_test_table_5 [id=04f8dfb749c04486895d6b0f7c206446]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.170919 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 55cf34de74694f47ad3f9f4d9bdffae4 (DEFAULT_TABLE table=rebalance_test_table_5 [id=04f8dfb749c04486895d6b0f7c206446]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.171204 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc77a7da906b419085258f72667ff185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.171586 30921 tablet_service.cc:1467] Processing CreateTablet for tablet d6b9a10c474f46bda0310ce661022b42 (DEFAULT_TABLE table=rebalance_test_table_5 [id=04f8dfb749c04486895d6b0f7c206446]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.171957 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b9a10c474f46bda0310ce661022b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.171976 32365 tablet_bootstrap.cc:492] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:38.172104 32365 ts_tablet_manager.cc:1367] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240503 14:13:38.173100 32365 raft_consensus.cc:348] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:38.173302 32365 raft_consensus.cc:374] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.173386 32365 raft_consensus.cc:729] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.173657 32365 consensus_queue.cc:260] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:38.175122 32365 ts_tablet_manager.cc:1398] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:38.176467 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55cf34de74694f47ad3f9f4d9bdffae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.181610 31181 tablet_service.cc:1467] Processing CreateTablet for tablet d6b9a10c474f46bda0310ce661022b42 (DEFAULT_TABLE table=rebalance_test_table_5 [id=04f8dfb749c04486895d6b0f7c206446]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.182068 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b9a10c474f46bda0310ce661022b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.182232 32370 tablet_bootstrap.cc:492] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:38.185034 32370 tablet_bootstrap.cc:654] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.186486 32364 tablet_bootstrap.cc:492] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:38.186730 32364 ts_tablet_manager.cc:1367] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.009s
I20240503 14:13:38.181094 32366 tablet_bootstrap.cc:492] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:38.187510 32366 ts_tablet_manager.cc:1367] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.005s
I20240503 14:13:38.188246 32364 raft_consensus.cc:348] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:38.188503 32366 raft_consensus.cc:348] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:38.188650 32364 raft_consensus.cc:374] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.188745 32366 raft_consensus.cc:374] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.188867 32364 raft_consensus.cc:729] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.188959 32366 raft_consensus.cc:729] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.189106 32364 consensus_queue.cc:260] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:38.189287 32366 consensus_queue.cc:260] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:38.189827 32364 ts_tablet_manager.cc:1398] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:38.190086 32364 tablet_bootstrap.cc:492] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:38.191325 32366 ts_tablet_manager.cc:1398] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:38.191725 32366 tablet_bootstrap.cc:492] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:38.193991 32366 tablet_bootstrap.cc:654] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.195794 32364 tablet_bootstrap.cc:654] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.197086 32366 tablet_bootstrap.cc:492] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:38.197244 32366 ts_tablet_manager.cc:1367] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 14:13:38.197628 32364 tablet_bootstrap.cc:492] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:38.197782 32364 ts_tablet_manager.cc:1367] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20240503 14:13:38.197867 32370 tablet_bootstrap.cc:492] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:38.198030 32370 ts_tablet_manager.cc:1367] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.010s
I20240503 14:13:38.198045 32366 raft_consensus.cc:348] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:38.198273 32366 raft_consensus.cc:374] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.198349 32366 raft_consensus.cc:729] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.198523 32364 raft_consensus.cc:348] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:38.198686 32364 raft_consensus.cc:374] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.198590 32366 consensus_queue.cc:260] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:38.198732 32364 raft_consensus.cc:729] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.198935 32364 consensus_queue.cc:260] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:38.198953 32370 raft_consensus.cc:348] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:38.199149 32370 raft_consensus.cc:374] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.199151 32366 ts_tablet_manager.cc:1398] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:38.199239 32370 raft_consensus.cc:729] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.199483 32370 consensus_queue.cc:260] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:38.200039 32370 ts_tablet_manager.cc:1398] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:38.200388 32370 tablet_bootstrap.cc:492] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:38.201193 32364 ts_tablet_manager.cc:1398] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:38.202061 32370 tablet_bootstrap.cc:654] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.203915 32364 tablet_bootstrap.cc:492] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:38.206650 32364 tablet_bootstrap.cc:654] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.208599 32364 tablet_bootstrap.cc:492] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:38.208676 32370 tablet_bootstrap.cc:492] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:38.208748 32364 ts_tablet_manager.cc:1367] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 14:13:38.208858 32370 ts_tablet_manager.cc:1367] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240503 14:13:38.209383 32364 raft_consensus.cc:348] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:38.209460 32370 raft_consensus.cc:348] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:38.209559 32364 raft_consensus.cc:374] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.209654 32364 raft_consensus.cc:729] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.209652 32370 raft_consensus.cc:374] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.209715 32370 raft_consensus.cc:729] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.209903 32364 consensus_queue.cc:260] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:38.209980 32370 consensus_queue.cc:260] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:38.210407 32364 ts_tablet_manager.cc:1398] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:38.210587 32370 ts_tablet_manager.cc:1398] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:38.210848 32370 tablet_bootstrap.cc:492] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:38.212500 32370 tablet_bootstrap.cc:654] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.214622 32370 tablet_bootstrap.cc:492] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:38.214758 32370 ts_tablet_manager.cc:1367] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:38.215479 32370 raft_consensus.cc:348] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:38.215699 32370 raft_consensus.cc:374] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.215798 32370 raft_consensus.cc:729] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.216030 32370 consensus_queue.cc:260] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:38.216537 32370 ts_tablet_manager.cc:1398] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:38.249181 32340 raft_consensus.cc:482] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:38.249377 32340 raft_consensus.cc:504] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:38.250284 32340 leader_election.cc:288] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:38.250818 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc77a7da906b419085258f72667ff185" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" is_pre_election: true
I20240503 14:13:38.250859 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc77a7da906b419085258f72667ff185" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:38.251088 31201 raft_consensus.cc:2454] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 0.
I20240503 14:13:38.251140 30939 raft_consensus.cc:2454] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 0.
I20240503 14:13:38.251744 31266 leader_election.cc:302] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 82cfdc130877418dbe9d980d368973be, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:38.252138 32340 raft_consensus.cc:2781] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:38.252282 32340 raft_consensus.cc:482] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:38.252373 32340 raft_consensus.cc:3037] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.254289 32340 raft_consensus.cc:504] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:38.254838 32340 leader_election.cc:288] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 election: Requested vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:38.255287 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc77a7da906b419085258f72667ff185" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:38.255273 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc77a7da906b419085258f72667ff185" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce"
I20240503 14:13:38.255492 31201 raft_consensus.cc:3037] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.255481 30940 raft_consensus.cc:3037] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.258306 31201 raft_consensus.cc:2454] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 1.
I20240503 14:13:38.258531 30940 raft_consensus.cc:2454] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 1.
I20240503 14:13:38.258736 31266 leader_election.cc:302] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82cfdc130877418dbe9d980d368973be, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:38.259030 32340 raft_consensus.cc:2781] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:38.259191 32340 raft_consensus.cc:686] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Becoming Leader. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Running, Role: LEADER
I20240503 14:13:38.259449 32340 consensus_queue.cc:237] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:38.262488 30798 catalog_manager.cc:5484] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be reported cstate change: term changed from 0 to 1, leader changed from <none> to 82cfdc130877418dbe9d980d368973be (127.22.242.68). New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:38.277287 32340 raft_consensus.cc:482] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:38.277459 32340 raft_consensus.cc:504] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:38.277947 32340 leader_election.cc:288] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:38.278390 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55cf34de74694f47ad3f9f4d9bdffae4" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" is_pre_election: true
I20240503 14:13:38.278422 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55cf34de74694f47ad3f9f4d9bdffae4" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:38.278584 31201 raft_consensus.cc:2454] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 0.
I20240503 14:13:38.278733 30940 raft_consensus.cc:2454] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 0.
I20240503 14:13:38.279079 31266 leader_election.cc:302] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82cfdc130877418dbe9d980d368973be, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:38.279358 32340 raft_consensus.cc:2781] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:38.279438 32340 raft_consensus.cc:482] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:38.279546 32340 raft_consensus.cc:3037] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.281414 32340 raft_consensus.cc:504] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:38.281944 32340 leader_election.cc:288] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 election: Requested vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:38.282336 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55cf34de74694f47ad3f9f4d9bdffae4" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:38.282402 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55cf34de74694f47ad3f9f4d9bdffae4" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce"
I20240503 14:13:38.282549 30940 raft_consensus.cc:3037] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.282583 31201 raft_consensus.cc:3037] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.284466 31201 raft_consensus.cc:2454] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 1.
I20240503 14:13:38.284492 30940 raft_consensus.cc:2454] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 1.
I20240503 14:13:38.284813 31266 leader_election.cc:302] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82cfdc130877418dbe9d980d368973be, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:38.285092 32340 raft_consensus.cc:2781] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:38.285236 32340 raft_consensus.cc:686] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Becoming Leader. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Running, Role: LEADER
I20240503 14:13:38.285478 32340 consensus_queue.cc:237] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:38.288120 30798 catalog_manager.cc:5484] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be reported cstate change: term changed from 0 to 1, leader changed from <none> to 82cfdc130877418dbe9d980d368973be (127.22.242.68). New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:38.341045 32349 raft_consensus.cc:482] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:38.341238 32349 raft_consensus.cc:504] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:38.341842 32349 leader_election.cc:288] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:38.342322 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b9a10c474f46bda0310ce661022b42" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:38.342340 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b9a10c474f46bda0310ce661022b42" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" is_pre_election: true
I20240503 14:13:38.342530 30940 raft_consensus.cc:2454] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:38.342554 31201 raft_consensus.cc:2454] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:38.342911 31005 leader_election.cc:302] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 6dc4948ed7c740ab89b9e349d151b0e5; no voters:
I20240503 14:13:38.343209 32349 raft_consensus.cc:2781] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:38.343324 32349 raft_consensus.cc:482] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:38.343410 32349 raft_consensus.cc:3037] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.345350 32349 raft_consensus.cc:504] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:38.345854 32349 leader_election.cc:288] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Requested vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:38.346320 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b9a10c474f46bda0310ce661022b42" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce"
I20240503 14:13:38.346547 31201 raft_consensus.cc:3037] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.349159 31201 raft_consensus.cc:2454] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:38.349527 31006 leader_election.cc:302] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:38.349509 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b9a10c474f46bda0310ce661022b42" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:38.349714 30940 raft_consensus.cc:3037] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.349809 32349 raft_consensus.cc:2781] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:38.349969 32349 raft_consensus.cc:686] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Becoming Leader. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Running, Role: LEADER
I20240503 14:13:38.350227 32349 consensus_queue.cc:237] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:38.351943 30940 raft_consensus.cc:2454] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:38.353245 30798 catalog_manager.cc:5484] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:38.663583 32340 consensus_queue.cc:1035] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.673923 32340 consensus_queue.cc:1035] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.699060 32384 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.721539 32340 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:38.755200 32380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:38.755534 32380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:13:38.760104 32349 consensus_queue.cc:1035] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:38.764941 32380 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:38.774873 32299 consensus_queue.cc:1035] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
INSERT report
rows total: 2000
time total: 253.824 ms
time per row: 0.126912 ms
I20240503 14:13:39.170264 30799 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36072:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:39.178503 31050 tablet_service.cc:1467] Processing CreateTablet for tablet 6234489254fc4882a39b7df28ddf673f (DEFAULT_TABLE table=rebalance_test_table_6 [id=08c2ccf24fcc40f9a0fd947d52b50f8e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.178828 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 6234489254fc4882a39b7df28ddf673f (DEFAULT_TABLE table=rebalance_test_table_6 [id=08c2ccf24fcc40f9a0fd947d52b50f8e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.179018 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6234489254fc4882a39b7df28ddf673f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.179016 31312 tablet_service.cc:1467] Processing CreateTablet for tablet 6234489254fc4882a39b7df28ddf673f (DEFAULT_TABLE table=rebalance_test_table_6 [id=08c2ccf24fcc40f9a0fd947d52b50f8e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.179272 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6234489254fc4882a39b7df28ddf673f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.179379 31312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6234489254fc4882a39b7df28ddf673f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.181020 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 589fce6d925b43d895588bac355135b0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=08c2ccf24fcc40f9a0fd947d52b50f8e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.181402 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 589fce6d925b43d895588bac355135b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.185019 31181 tablet_service.cc:1467] Processing CreateTablet for tablet c837def6d76242f08b20a448541e64d6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=08c2ccf24fcc40f9a0fd947d52b50f8e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.185423 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c837def6d76242f08b20a448541e64d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.189591 31310 tablet_service.cc:1467] Processing CreateTablet for tablet c837def6d76242f08b20a448541e64d6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=08c2ccf24fcc40f9a0fd947d52b50f8e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.189827 31311 tablet_service.cc:1467] Processing CreateTablet for tablet 589fce6d925b43d895588bac355135b0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=08c2ccf24fcc40f9a0fd947d52b50f8e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.190085 31310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c837def6d76242f08b20a448541e64d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.190289 31311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 589fce6d925b43d895588bac355135b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.193325 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 589fce6d925b43d895588bac355135b0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=08c2ccf24fcc40f9a0fd947d52b50f8e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.193435 31052 tablet_service.cc:1467] Processing CreateTablet for tablet c837def6d76242f08b20a448541e64d6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=08c2ccf24fcc40f9a0fd947d52b50f8e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.194051 32452 tablet_bootstrap.cc:492] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:39.195065 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 589fce6d925b43d895588bac355135b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.196437 32452 tablet_bootstrap.cc:654] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.198525 32450 tablet_bootstrap.cc:492] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:39.199802 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c837def6d76242f08b20a448541e64d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.202363 32452 tablet_bootstrap.cc:492] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:39.202556 32452 ts_tablet_manager.cc:1367] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20240503 14:13:39.203400 32452 raft_consensus.cc:348] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:39.203724 32452 raft_consensus.cc:374] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.203955 32452 raft_consensus.cc:729] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.204264 32452 consensus_queue.cc:260] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:39.205358 32451 tablet_bootstrap.cc:492] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:39.205796 32452 ts_tablet_manager.cc:1398] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:39.206826 32451 tablet_bootstrap.cc:654] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.209399 32450 tablet_bootstrap.cc:654] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.211114 32451 tablet_bootstrap.cc:492] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:39.211313 32451 ts_tablet_manager.cc:1367] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 14:13:39.212183 32451 raft_consensus.cc:348] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:39.212502 32451 raft_consensus.cc:374] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.212663 32451 raft_consensus.cc:729] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.212970 32451 consensus_queue.cc:260] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:39.213726 32451 ts_tablet_manager.cc:1398] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:39.215138 32452 tablet_bootstrap.cc:492] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:39.215993 32451 tablet_bootstrap.cc:492] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:39.216849 32452 tablet_bootstrap.cc:654] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.217720 32454 tablet_bootstrap.cc:492] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:39.217767 32450 tablet_bootstrap.cc:492] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:39.217911 32450 ts_tablet_manager.cc:1367] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240503 14:13:39.219120 32450 raft_consensus.cc:348] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:39.219380 32450 raft_consensus.cc:374] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.219516 32450 raft_consensus.cc:729] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.219764 32454 tablet_bootstrap.cc:654] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.219839 32450 consensus_queue.cc:260] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:39.222071 32451 tablet_bootstrap.cc:654] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.224777 32452 tablet_bootstrap.cc:492] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:39.225106 32452 ts_tablet_manager.cc:1367] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.007s
I20240503 14:13:39.225952 32450 ts_tablet_manager.cc:1398] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:13:39.226128 32452 raft_consensus.cc:348] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:39.226436 32452 raft_consensus.cc:374] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.226521 32452 raft_consensus.cc:729] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.226851 32452 consensus_queue.cc:260] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:39.227876 32451 tablet_bootstrap.cc:492] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:39.228252 32452 ts_tablet_manager.cc:1398] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:39.229091 32451 ts_tablet_manager.cc:1367] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.006s
I20240503 14:13:39.230021 32450 tablet_bootstrap.cc:492] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:39.230118 32451 raft_consensus.cc:348] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:39.230595 32451 raft_consensus.cc:374] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.230726 32451 raft_consensus.cc:729] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.231062 32451 consensus_queue.cc:260] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:39.231956 32451 ts_tablet_manager.cc:1398] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:39.232247 32451 tablet_bootstrap.cc:492] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:39.233546 32450 tablet_bootstrap.cc:654] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.234942 32451 tablet_bootstrap.cc:654] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.239560 32450 tablet_bootstrap.cc:492] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:39.239750 32450 ts_tablet_manager.cc:1367] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240503 14:13:39.240693 32450 raft_consensus.cc:348] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:39.240917 32450 raft_consensus.cc:374] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.240990 32450 raft_consensus.cc:729] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.241214 32450 consensus_queue.cc:260] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:39.241720 32450 ts_tablet_manager.cc:1398] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:39.242219 32450 tablet_bootstrap.cc:492] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:39.244131 32450 tablet_bootstrap.cc:654] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.244683 32451 tablet_bootstrap.cc:492] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:39.244875 32451 ts_tablet_manager.cc:1367] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240503 14:13:39.245709 32451 raft_consensus.cc:348] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:39.246042 32451 raft_consensus.cc:374] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.246198 32451 raft_consensus.cc:729] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.246488 32451 consensus_queue.cc:260] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:39.246985 32451 ts_tablet_manager.cc:1398] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:39.253054 32454 tablet_bootstrap.cc:492] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:39.253085 32450 tablet_bootstrap.cc:492] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:39.253209 32454 ts_tablet_manager.cc:1367] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.004s
I20240503 14:13:39.253213 32450 ts_tablet_manager.cc:1367] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240503 14:13:39.253932 32450 raft_consensus.cc:348] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:39.253983 32454 raft_consensus.cc:348] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:39.254096 32450 raft_consensus.cc:374] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.254180 32454 raft_consensus.cc:374] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.254196 32450 raft_consensus.cc:729] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.254264 32454 raft_consensus.cc:729] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.254477 32450 consensus_queue.cc:260] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:39.254472 32454 consensus_queue.cc:260] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:39.254982 32454 ts_tablet_manager.cc:1398] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:39.255584 32450 ts_tablet_manager.cc:1398] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:39.285122 32299 raft_consensus.cc:482] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:39.285351 32299 raft_consensus.cc:504] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:39.286136 32299 leader_election.cc:288] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:39.286677 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589fce6d925b43d895588bac355135b0" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:39.286731 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589fce6d925b43d895588bac355135b0" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be" is_pre_election: true
I20240503 14:13:39.286926 30940 raft_consensus.cc:2454] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:39.286937 31331 raft_consensus.cc:2454] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:39.287331 31005 leader_election.cc:302] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 6dc4948ed7c740ab89b9e349d151b0e5; no voters:
I20240503 14:13:39.287613 32299 raft_consensus.cc:2781] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:39.287703 32299 raft_consensus.cc:482] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:39.287812 32299 raft_consensus.cc:3037] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.289848 32299 raft_consensus.cc:504] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:39.290388 32299 leader_election.cc:288] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Requested vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:39.290814 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589fce6d925b43d895588bac355135b0" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:39.290834 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589fce6d925b43d895588bac355135b0" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be"
I20240503 14:13:39.291041 31331 raft_consensus.cc:3037] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.291049 30940 raft_consensus.cc:3037] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.293833 30940 raft_consensus.cc:2454] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:39.293851 31331 raft_consensus.cc:2454] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:39.294232 31007 leader_election.cc:302] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, 82cfdc130877418dbe9d980d368973be; no voters:
I20240503 14:13:39.294499 32299 raft_consensus.cc:2781] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:39.294653 32299 raft_consensus.cc:686] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Becoming Leader. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Running, Role: LEADER
I20240503 14:13:39.294914 32299 consensus_queue.cc:237] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:39.297955 30794 catalog_manager.cc:5484] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:39.303500 32292 raft_consensus.cc:482] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:39.303653 32292 raft_consensus.cc:504] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:39.304366 32292 leader_election.cc:288] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:39.304776 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6234489254fc4882a39b7df28ddf673f" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" is_pre_election: true
I20240503 14:13:39.304860 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6234489254fc4882a39b7df28ddf673f" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:39.305009 31071 raft_consensus.cc:2454] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 0.
I20240503 14:13:39.305063 30940 raft_consensus.cc:2454] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 0.
I20240503 14:13:39.305438 31268 leader_election.cc:302] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, 82cfdc130877418dbe9d980d368973be; no voters:
I20240503 14:13:39.305680 32292 raft_consensus.cc:2781] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:39.305794 32292 raft_consensus.cc:482] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:39.305891 32292 raft_consensus.cc:3037] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.307685 32292 raft_consensus.cc:504] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:39.308252 32292 leader_election.cc:288] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 election: Requested vote from peers 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:39.308619 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6234489254fc4882a39b7df28ddf673f" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5"
I20240503 14:13:39.308777 31071 raft_consensus.cc:3037] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.308720 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6234489254fc4882a39b7df28ddf673f" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:39.308914 30940 raft_consensus.cc:3037] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.310595 31071 raft_consensus.cc:2454] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 1.
I20240503 14:13:39.310808 30940 raft_consensus.cc:2454] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 1.
I20240503 14:13:39.310978 31268 leader_election.cc:302] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, 82cfdc130877418dbe9d980d368973be; no voters:
I20240503 14:13:39.311287 32292 raft_consensus.cc:2781] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:39.311403 32292 raft_consensus.cc:686] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Becoming Leader. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Running, Role: LEADER
I20240503 14:13:39.311686 32292 consensus_queue.cc:237] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:39.314308 30794 catalog_manager.cc:5484] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be reported cstate change: term changed from 0 to 1, leader changed from <none> to 82cfdc130877418dbe9d980d368973be (127.22.242.68). New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:39.350638 32299 raft_consensus.cc:482] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:39.350791 32299 raft_consensus.cc:504] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:39.351495 32299 leader_election.cc:288] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:39.351804 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c837def6d76242f08b20a448541e64d6" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" is_pre_election: true
I20240503 14:13:39.352022 31201 raft_consensus.cc:2454] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:39.351976 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c837def6d76242f08b20a448541e64d6" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be" is_pre_election: true
I20240503 14:13:39.352221 31331 raft_consensus.cc:2454] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:39.352460 31006 leader_election.cc:302] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:39.352737 32299 raft_consensus.cc:2781] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:39.352867 32299 raft_consensus.cc:482] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:39.352958 32299 raft_consensus.cc:3037] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.355274 32299 raft_consensus.cc:504] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:39.355806 32299 leader_election.cc:288] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Requested vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:39.356141 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c837def6d76242f08b20a448541e64d6" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce"
I20240503 14:13:39.356263 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c837def6d76242f08b20a448541e64d6" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be"
I20240503 14:13:39.356377 31201 raft_consensus.cc:3037] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.356473 31331 raft_consensus.cc:3037] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.359128 31331 raft_consensus.cc:2454] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:39.359127 31201 raft_consensus.cc:2454] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:39.359485 31006 leader_election.cc:302] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:39.359767 32299 raft_consensus.cc:2781] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:39.360033 32299 raft_consensus.cc:686] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Becoming Leader. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Running, Role: LEADER
I20240503 14:13:39.360328 32299 consensus_queue.cc:237] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:39.363322 30799 catalog_manager.cc:5484] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:39.679934 32299 consensus_queue.cc:1035] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:39.686216 32299 consensus_queue.cc:1035] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:39.729519 32466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:39.729859 32466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:39.737814 32466 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 14:13:39.873210 32292 consensus_queue.cc:1035] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:39.877097 32292 consensus_queue.cc:1035] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:39.877071 32299 consensus_queue.cc:1035] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:39.940541 31137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.68:37063, user_credentials={real_user=slave}} blocked reactor thread for 74710.2us
I20240503 14:13:39.945600 31201 raft_consensus.cc:1261] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:39.948673 32396 consensus_queue.cc:1035] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 290.686 ms
time per row: 0.145343 ms
-----------------------------------------------------
Suppressions used:
count bytes template
12 3028 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:13:41.131423 30796 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36072:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:41.141181 31052 tablet_service.cc:1467] Processing CreateTablet for tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87 (DEFAULT_TABLE table=rebalance_test_table_7 [id=5b67f0a6a8d34286994597065f21a92b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.141505 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87 (DEFAULT_TABLE table=rebalance_test_table_7 [id=5b67f0a6a8d34286994597065f21a92b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.141659 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.141911 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.142233 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87 (DEFAULT_TABLE table=rebalance_test_table_7 [id=5b67f0a6a8d34286994597065f21a92b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.142539 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.144610 31051 tablet_service.cc:1467] Processing CreateTablet for tablet c86ee72320b647eb84e376f8995b55d2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=5b67f0a6a8d34286994597065f21a92b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.144964 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86ee72320b647eb84e376f8995b55d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.145447 31182 tablet_service.cc:1467] Processing CreateTablet for tablet c86ee72320b647eb84e376f8995b55d2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=5b67f0a6a8d34286994597065f21a92b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.145876 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86ee72320b647eb84e376f8995b55d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.148486 31180 tablet_service.cc:1467] Processing CreateTablet for tablet 0f45824b1fff4302abd03c1fb48c6c7b (DEFAULT_TABLE table=rebalance_test_table_7 [id=5b67f0a6a8d34286994597065f21a92b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.148484 31311 tablet_service.cc:1467] Processing CreateTablet for tablet c86ee72320b647eb84e376f8995b55d2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=5b67f0a6a8d34286994597065f21a92b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.148835 31180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f45824b1fff4302abd03c1fb48c6c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.148859 31311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86ee72320b647eb84e376f8995b55d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.149005 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 0f45824b1fff4302abd03c1fb48c6c7b (DEFAULT_TABLE table=rebalance_test_table_7 [id=5b67f0a6a8d34286994597065f21a92b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.149505 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f45824b1fff4302abd03c1fb48c6c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.153342 31310 tablet_service.cc:1467] Processing CreateTablet for tablet 0f45824b1fff4302abd03c1fb48c6c7b (DEFAULT_TABLE table=rebalance_test_table_7 [id=5b67f0a6a8d34286994597065f21a92b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.153702 31310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f45824b1fff4302abd03c1fb48c6c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.158317 32535 tablet_bootstrap.cc:492] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:41.159915 32535 tablet_bootstrap.cc:654] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.164875 32536 tablet_bootstrap.cc:492] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:41.166868 32534 tablet_bootstrap.cc:492] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:41.167083 32535 tablet_bootstrap.cc:492] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:41.167142 32536 tablet_bootstrap.cc:654] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.167479 32535 ts_tablet_manager.cc:1367] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 14:13:41.168073 32537 tablet_bootstrap.cc:492] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:41.168346 32535 raft_consensus.cc:348] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:41.168656 32535 raft_consensus.cc:374] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.168753 32534 tablet_bootstrap.cc:654] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.168807 32535 raft_consensus.cc:729] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.169122 32535 consensus_queue.cc:260] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:41.169829 32535 ts_tablet_manager.cc:1398] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:41.170122 32535 tablet_bootstrap.cc:492] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:41.171608 32535 tablet_bootstrap.cc:654] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.179430 32535 tablet_bootstrap.cc:492] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:41.179670 32535 ts_tablet_manager.cc:1367] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240503 14:13:41.179953 32537 tablet_bootstrap.cc:654] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.180543 32535 raft_consensus.cc:348] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:41.180814 32535 raft_consensus.cc:374] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.180924 32535 raft_consensus.cc:729] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.181226 32535 consensus_queue.cc:260] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:41.181807 32535 ts_tablet_manager.cc:1398] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:41.183807 32536 tablet_bootstrap.cc:492] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:41.184253 32536 ts_tablet_manager.cc:1367] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20240503 14:13:41.183109 32537 tablet_bootstrap.cc:492] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:41.184739 32537 ts_tablet_manager.cc:1367] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240503 14:13:41.185387 32534 tablet_bootstrap.cc:492] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:41.185586 32534 ts_tablet_manager.cc:1367] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20240503 14:13:41.185732 32536 raft_consensus.cc:348] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:41.186110 32536 raft_consensus.cc:374] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.186067 32537 raft_consensus.cc:348] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:41.186306 32537 raft_consensus.cc:374] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.186305 32536 raft_consensus.cc:729] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.186442 32537 raft_consensus.cc:729] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.186465 32534 raft_consensus.cc:348] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:41.186697 32534 raft_consensus.cc:374] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.186797 32534 raft_consensus.cc:729] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.186707 32537 consensus_queue.cc:260] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:41.186863 32536 consensus_queue.cc:260] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:41.187004 32534 consensus_queue.cc:260] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:41.187263 32537 ts_tablet_manager.cc:1398] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:41.187552 32534 ts_tablet_manager.cc:1398] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:41.187659 32537 tablet_bootstrap.cc:492] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:41.187875 32534 tablet_bootstrap.cc:492] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:41.189540 32534 tablet_bootstrap.cc:654] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.189621 32536 ts_tablet_manager.cc:1398] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:41.189942 32536 tablet_bootstrap.cc:492] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:41.192242 32536 tablet_bootstrap.cc:654] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.193866 32537 tablet_bootstrap.cc:654] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.194530 32534 tablet_bootstrap.cc:492] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:41.194722 32534 ts_tablet_manager.cc:1367] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:13:41.195812 32534 raft_consensus.cc:348] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:41.196092 32534 raft_consensus.cc:374] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.196251 32534 raft_consensus.cc:729] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.196595 32534 consensus_queue.cc:260] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:41.196822 32536 tablet_bootstrap.cc:492] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:41.196992 32536 ts_tablet_manager.cc:1367] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:13:41.197183 32534 ts_tablet_manager.cc:1398] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:41.197728 32536 raft_consensus.cc:348] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:41.197940 32536 raft_consensus.cc:374] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.198045 32536 raft_consensus.cc:729] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.198302 32536 consensus_queue.cc:260] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:41.198886 32536 ts_tablet_manager.cc:1398] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:41.197585 32534 tablet_bootstrap.cc:492] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:41.200634 32534 tablet_bootstrap.cc:654] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.203337 32534 tablet_bootstrap.cc:492] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:41.203475 32534 ts_tablet_manager.cc:1367] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:13:41.203493 32537 tablet_bootstrap.cc:492] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:41.203653 32537 ts_tablet_manager.cc:1367] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s
I20240503 14:13:41.204051 32534 raft_consensus.cc:348] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:41.204208 32534 raft_consensus.cc:374] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.204303 32534 raft_consensus.cc:729] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.204239 32537 raft_consensus.cc:348] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:41.204433 32537 raft_consensus.cc:374] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.204504 32537 raft_consensus.cc:729] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.204511 32534 consensus_queue.cc:260] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:41.204731 32537 consensus_queue.cc:260] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:41.205096 32534 ts_tablet_manager.cc:1398] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:41.205372 32537 ts_tablet_manager.cc:1398] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:41.241215 32507 raft_consensus.cc:482] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:41.241401 32507 raft_consensus.cc:504] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:41.242121 32507 leader_election.cc:288] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:41.242584 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f45824b1fff4302abd03c1fb48c6c7b" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:41.242714 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f45824b1fff4302abd03c1fb48c6c7b" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" is_pre_election: true
I20240503 14:13:41.242810 30940 raft_consensus.cc:2454] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 0.
I20240503 14:13:41.242964 31201 raft_consensus.cc:2454] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 0.
I20240503 14:13:41.243302 31265 leader_election.cc:302] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 82cfdc130877418dbe9d980d368973be; no voters:
I20240503 14:13:41.243618 32507 raft_consensus.cc:2781] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:41.243726 32507 raft_consensus.cc:482] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:41.243796 32507 raft_consensus.cc:3037] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.245692 32507 raft_consensus.cc:504] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:41.246304 32507 leader_election.cc:288] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 election: Requested vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:41.246649 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f45824b1fff4302abd03c1fb48c6c7b" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:41.246819 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f45824b1fff4302abd03c1fb48c6c7b" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce"
I20240503 14:13:41.246886 30940 raft_consensus.cc:3037] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.247026 31201 raft_consensus.cc:3037] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.248956 30940 raft_consensus.cc:2454] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 1.
I20240503 14:13:41.249799 31201 raft_consensus.cc:2454] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 1.
I20240503 14:13:41.250114 31265 leader_election.cc:302] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 82cfdc130877418dbe9d980d368973be; no voters:
I20240503 14:13:41.250404 32507 raft_consensus.cc:2781] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:41.250775 32507 raft_consensus.cc:686] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Becoming Leader. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Running, Role: LEADER
I20240503 14:13:41.251236 32507 consensus_queue.cc:237] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:41.254138 30798 catalog_manager.cc:5484] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be reported cstate change: term changed from 0 to 1, leader changed from <none> to 82cfdc130877418dbe9d980d368973be (127.22.242.68). New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:41.280270 32529 raft_consensus.cc:482] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:41.280455 32529 raft_consensus.cc:504] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:41.281270 32529 leader_election.cc:288] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:41.281563 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86ee72320b647eb84e376f8995b55d2" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" is_pre_election: true
I20240503 14:13:41.281787 31201 raft_consensus.cc:2454] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:41.281755 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86ee72320b647eb84e376f8995b55d2" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be" is_pre_election: true
I20240503 14:13:41.281978 31331 raft_consensus.cc:2454] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:41.282182 31006 leader_election.cc:302] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:41.282454 32529 raft_consensus.cc:2781] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:41.282578 32529 raft_consensus.cc:482] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:41.282661 32529 raft_consensus.cc:3037] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.284525 32529 raft_consensus.cc:504] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:41.285018 32529 leader_election.cc:288] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Requested vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:41.285377 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86ee72320b647eb84e376f8995b55d2" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce"
I20240503 14:13:41.285583 31201 raft_consensus.cc:3037] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.285521 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86ee72320b647eb84e376f8995b55d2" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be"
I20240503 14:13:41.285717 31331 raft_consensus.cc:3037] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.288482 31331 raft_consensus.cc:2454] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:41.288475 31201 raft_consensus.cc:2454] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:41.288997 31007 leader_election.cc:302] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, 82cfdc130877418dbe9d980d368973be; no voters:
I20240503 14:13:41.289300 32529 raft_consensus.cc:2781] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:41.289451 32529 raft_consensus.cc:686] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Becoming Leader. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Running, Role: LEADER
I20240503 14:13:41.289716 32529 consensus_queue.cc:237] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:41.292904 30798 catalog_manager.cc:5484] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:41.439180 32529 raft_consensus.cc:482] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:41.439420 32529 raft_consensus.cc:504] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:41.440151 32529 leader_election.cc:288] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:41.440639 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bd38c5cccf34dc3b8e6ec5425cb5e87" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" is_pre_election: true
I20240503 14:13:41.440757 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bd38c5cccf34dc3b8e6ec5425cb5e87" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:41.440863 31201 raft_consensus.cc:2454] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:41.440984 30940 raft_consensus.cc:2454] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:41.441314 31006 leader_election.cc:302] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc4948ed7c740ab89b9e349d151b0e5, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:41.441588 32529 raft_consensus.cc:2781] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:41.441686 32529 raft_consensus.cc:482] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:41.441799 32529 raft_consensus.cc:3037] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.444025 32529 raft_consensus.cc:504] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:41.444643 32529 leader_election.cc:288] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Requested vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:41.445099 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bd38c5cccf34dc3b8e6ec5425cb5e87" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:41.445286 30940 raft_consensus.cc:3037] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.445307 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bd38c5cccf34dc3b8e6ec5425cb5e87" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce"
I20240503 14:13:41.445521 31201 raft_consensus.cc:3037] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.447140 30940 raft_consensus.cc:2454] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:41.447369 31201 raft_consensus.cc:2454] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:41.447525 31005 leader_election.cc:302] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 6dc4948ed7c740ab89b9e349d151b0e5; no voters:
I20240503 14:13:41.447830 32529 raft_consensus.cc:2781] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:41.447993 32529 raft_consensus.cc:686] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Becoming Leader. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Running, Role: LEADER
I20240503 14:13:41.448279 32529 consensus_queue.cc:237] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } }
I20240503 14:13:41.451220 30798 catalog_manager.cc:5484] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:41.668570 32507 consensus_queue.cc:1035] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:41.671133 32529 consensus_queue.cc:1035] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:41.687086 32528 consensus_queue.cc:1035] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:41.689322 32556 consensus_queue.cc:1035] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:41.823055 32551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:41.823407 32551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:41.830806 32551 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:41.856098 32529 consensus_queue.cc:1035] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:41.860437 32528 consensus_queue.cc:1035] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
INSERT report
rows total: 2000
time total: 214.417 ms
time per row: 0.107209 ms
I20240503 14:13:42.210703 30794 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36072:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:42.219255 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 584769402a7e4e8a985a8f2363e2b8b9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c243ced42cda4e3996901964941b50f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.219404 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 584769402a7e4e8a985a8f2363e2b8b9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c243ced42cda4e3996901964941b50f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.219738 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584769402a7e4e8a985a8f2363e2b8b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.219877 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584769402a7e4e8a985a8f2363e2b8b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.221064 31180 tablet_service.cc:1467] Processing CreateTablet for tablet 69b741aad35d4bfc8fea5b8e21d4a733 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c243ced42cda4e3996901964941b50f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.221451 31180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b741aad35d4bfc8fea5b8e21d4a733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.221364 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 1a9b84d307b8432bbdf6b70a11bf66b9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c243ced42cda4e3996901964941b50f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.221724 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a9b84d307b8432bbdf6b70a11bf66b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.223505 31050 tablet_service.cc:1467] Processing CreateTablet for tablet 69b741aad35d4bfc8fea5b8e21d4a733 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c243ced42cda4e3996901964941b50f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.223902 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b741aad35d4bfc8fea5b8e21d4a733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.224658 31310 tablet_service.cc:1467] Processing CreateTablet for tablet 584769402a7e4e8a985a8f2363e2b8b9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c243ced42cda4e3996901964941b50f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.225064 31310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584769402a7e4e8a985a8f2363e2b8b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.227452 31052 tablet_service.cc:1467] Processing CreateTablet for tablet 1a9b84d307b8432bbdf6b70a11bf66b9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c243ced42cda4e3996901964941b50f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.227824 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a9b84d307b8432bbdf6b70a11bf66b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.228441 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 1a9b84d307b8432bbdf6b70a11bf66b9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c243ced42cda4e3996901964941b50f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.228545 31311 tablet_service.cc:1467] Processing CreateTablet for tablet 69b741aad35d4bfc8fea5b8e21d4a733 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c243ced42cda4e3996901964941b50f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.228912 31311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b741aad35d4bfc8fea5b8e21d4a733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.233101 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a9b84d307b8432bbdf6b70a11bf66b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.244385 32617 tablet_bootstrap.cc:492] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:42.247471 32617 tablet_bootstrap.cc:654] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.255659 32618 tablet_bootstrap.cc:492] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:42.256554 32619 tablet_bootstrap.cc:492] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:42.258354 32619 tablet_bootstrap.cc:654] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.258402 32618 tablet_bootstrap.cc:654] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.260067 32616 tablet_bootstrap.cc:492] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:42.263259 32616 tablet_bootstrap.cc:654] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.269769 32617 tablet_bootstrap.cc:492] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:42.269980 32617 ts_tablet_manager.cc:1367] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.000s
I20240503 14:13:42.271044 32617 raft_consensus.cc:348] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:42.271319 32617 raft_consensus.cc:374] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.271472 32617 raft_consensus.cc:729] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.271847 32617 consensus_queue.cc:260] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:42.273344 32619 tablet_bootstrap.cc:492] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:42.273605 32619 ts_tablet_manager.cc:1367] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.005s
I20240503 14:13:42.275458 32619 raft_consensus.cc:348] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:42.275869 32619 raft_consensus.cc:374] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.276046 32619 raft_consensus.cc:729] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.276427 32619 consensus_queue.cc:260] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:42.277299 32619 ts_tablet_manager.cc:1398] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:42.278173 32619 tablet_bootstrap.cc:492] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:42.280153 32617 ts_tablet_manager.cc:1398] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240503 14:13:42.280318 32619 tablet_bootstrap.cc:654] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.287446 32617 tablet_bootstrap.cc:492] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:42.298358 32617 tablet_bootstrap.cc:654] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.301043 32616 tablet_bootstrap.cc:492] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:42.301252 32616 ts_tablet_manager.cc:1367] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.041s user 0.014s sys 0.019s
I20240503 14:13:42.302032 32616 raft_consensus.cc:348] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:42.302316 32616 raft_consensus.cc:374] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.302361 32618 tablet_bootstrap.cc:492] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:42.302431 32616 raft_consensus.cc:729] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.302717 32616 consensus_queue.cc:260] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:42.302852 32618 ts_tablet_manager.cc:1367] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.031s
I20240503 14:13:42.303503 32616 ts_tablet_manager.cc:1398] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:42.303725 32618 raft_consensus.cc:348] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:42.303901 32616 tablet_bootstrap.cc:492] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:42.304069 32618 raft_consensus.cc:374] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.304271 32618 raft_consensus.cc:729] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.304605 32618 consensus_queue.cc:260] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:42.305490 32616 tablet_bootstrap.cc:654] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.305759 32618 ts_tablet_manager.cc:1398] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:13:42.306107 32618 tablet_bootstrap.cc:492] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:42.307965 32618 tablet_bootstrap.cc:654] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.315165 32618 tablet_bootstrap.cc:492] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:42.316092 32618 ts_tablet_manager.cc:1367] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.004s
I20240503 14:13:42.316589 32616 tablet_bootstrap.cc:492] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:42.316823 32616 ts_tablet_manager.cc:1367] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20240503 14:13:42.317466 32618 raft_consensus.cc:348] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:42.317584 32616 raft_consensus.cc:348] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:42.317902 32616 raft_consensus.cc:374] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.318006 32616 raft_consensus.cc:729] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.318409 32616 consensus_queue.cc:260] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:42.319033 32618 raft_consensus.cc:374] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.319067 32616 ts_tablet_manager.cc:1398] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:42.319293 32618 raft_consensus.cc:729] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.319499 32616 tablet_bootstrap.cc:492] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:42.319667 32618 consensus_queue.cc:260] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:42.320592 32618 ts_tablet_manager.cc:1398] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:42.322448 32619 tablet_bootstrap.cc:492] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:42.322666 32619 ts_tablet_manager.cc:1367] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.026s
I20240503 14:13:42.322690 32617 tablet_bootstrap.cc:492] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:42.322886 32617 ts_tablet_manager.cc:1367] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.009s
I20240503 14:13:42.323297 32616 tablet_bootstrap.cc:654] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.323468 32619 raft_consensus.cc:348] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:42.323705 32619 raft_consensus.cc:374] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.323805 32619 raft_consensus.cc:729] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.324056 32619 consensus_queue.cc:260] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:42.324602 32619 ts_tablet_manager.cc:1398] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:42.325774 32617 raft_consensus.cc:348] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:42.326356 32617 raft_consensus.cc:374] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.326494 32617 raft_consensus.cc:729] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.326848 32617 consensus_queue.cc:260] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:42.328483 32617 ts_tablet_manager.cc:1398] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:13:42.329782 32616 tablet_bootstrap.cc:492] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:42.329981 32616 ts_tablet_manager.cc:1367] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20240503 14:13:42.330740 32616 raft_consensus.cc:348] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:42.330893 32616 raft_consensus.cc:374] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.330987 32616 raft_consensus.cc:729] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.331228 32616 consensus_queue.cc:260] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:42.331698 32616 ts_tablet_manager.cc:1398] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:42.343981 32096 raft_consensus.cc:482] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:42.344141 32096 raft_consensus.cc:504] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:42.344863 32096 leader_election.cc:288] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:42.345206 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a9b84d307b8432bbdf6b70a11bf66b9" candidate_uuid: "0dc902d91ccc435a82a5ede4a7729811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" is_pre_election: true
I20240503 14:13:42.345289 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a9b84d307b8432bbdf6b70a11bf66b9" candidate_uuid: "0dc902d91ccc435a82a5ede4a7729811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" is_pre_election: true
I20240503 14:13:42.345407 31202 raft_consensus.cc:2454] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dc902d91ccc435a82a5ede4a7729811 in term 0.
I20240503 14:13:42.345518 31071 raft_consensus.cc:2454] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dc902d91ccc435a82a5ede4a7729811 in term 0.
I20240503 14:13:42.345809 30875 leader_election.cc:302] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:42.346081 32096 raft_consensus.cc:2781] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:42.346199 32096 raft_consensus.cc:482] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:42.346292 32096 raft_consensus.cc:3037] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.348057 32096 raft_consensus.cc:504] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:42.348626 32096 leader_election.cc:288] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [CANDIDATE]: Term 1 election: Requested vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:42.348999 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a9b84d307b8432bbdf6b70a11bf66b9" candidate_uuid: "0dc902d91ccc435a82a5ede4a7729811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce"
I20240503 14:13:42.349108 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a9b84d307b8432bbdf6b70a11bf66b9" candidate_uuid: "0dc902d91ccc435a82a5ede4a7729811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5"
I20240503 14:13:42.349200 31202 raft_consensus.cc:3037] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.349321 31071 raft_consensus.cc:3037] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.351255 31202 raft_consensus.cc:2454] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dc902d91ccc435a82a5ede4a7729811 in term 1.
I20240503 14:13:42.351369 31071 raft_consensus.cc:2454] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dc902d91ccc435a82a5ede4a7729811 in term 1.
I20240503 14:13:42.351693 30875 leader_election.cc:302] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:42.352017 32096 raft_consensus.cc:2781] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:42.352509 32096 raft_consensus.cc:686] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 LEADER]: Becoming Leader. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Running, Role: LEADER
I20240503 14:13:42.353013 32096 consensus_queue.cc:237] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:42.355235 32596 raft_consensus.cc:482] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:42.355374 32596 raft_consensus.cc:504] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:42.355929 30796 catalog_manager.cc:5484] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "0dc902d91ccc435a82a5ede4a7729811" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:42.356146 32596 leader_election.cc:288] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:42.356570 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584769402a7e4e8a985a8f2363e2b8b9" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:42.356776 30940 raft_consensus.cc:2454] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:42.356778 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584769402a7e4e8a985a8f2363e2b8b9" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be" is_pre_election: true
I20240503 14:13:42.357054 31331 raft_consensus.cc:2454] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:42.357177 31005 leader_election.cc:302] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 6dc4948ed7c740ab89b9e349d151b0e5; no voters:
I20240503 14:13:42.357458 32596 raft_consensus.cc:2781] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:42.357573 32596 raft_consensus.cc:482] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:42.357663 32596 raft_consensus.cc:3037] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.360051 32596 raft_consensus.cc:504] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:42.360579 32596 leader_election.cc:288] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Requested vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:42.361047 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584769402a7e4e8a985a8f2363e2b8b9" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:42.361056 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584769402a7e4e8a985a8f2363e2b8b9" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be"
I20240503 14:13:42.361239 31331 raft_consensus.cc:3037] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.361223 30940 raft_consensus.cc:3037] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.363013 30940 raft_consensus.cc:2454] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:42.363067 31331 raft_consensus.cc:2454] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:42.363422 31005 leader_election.cc:302] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 6dc4948ed7c740ab89b9e349d151b0e5; no voters:
I20240503 14:13:42.363660 32596 raft_consensus.cc:2781] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:42.363893 32596 raft_consensus.cc:686] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Becoming Leader. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Running, Role: LEADER
I20240503 14:13:42.364182 32596 consensus_queue.cc:237] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:42.367079 30796 catalog_manager.cc:5484] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:42.399756 32556 raft_consensus.cc:482] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:42.400007 32556 raft_consensus.cc:504] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:42.400743 32556 leader_election.cc:288] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:42.401121 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b741aad35d4bfc8fea5b8e21d4a733" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" is_pre_election: true
I20240503 14:13:42.401304 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b741aad35d4bfc8fea5b8e21d4a733" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" is_pre_election: true
I20240503 14:13:42.401369 31202 raft_consensus.cc:2454] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 0.
I20240503 14:13:42.401538 31071 raft_consensus.cc:2454] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 0.
I20240503 14:13:42.401758 31266 leader_election.cc:302] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82cfdc130877418dbe9d980d368973be, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:42.402053 32556 raft_consensus.cc:2781] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:42.402180 32556 raft_consensus.cc:482] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:42.402264 32556 raft_consensus.cc:3037] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.404438 32556 raft_consensus.cc:504] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:42.404978 32556 leader_election.cc:288] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 election: Requested vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:42.405336 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b741aad35d4bfc8fea5b8e21d4a733" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce"
I20240503 14:13:42.405542 31202 raft_consensus.cc:3037] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.405500 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b741aad35d4bfc8fea5b8e21d4a733" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4948ed7c740ab89b9e349d151b0e5"
I20240503 14:13:42.405696 31071 raft_consensus.cc:3037] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.408346 31202 raft_consensus.cc:2454] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 1.
I20240503 14:13:42.408452 31071 raft_consensus.cc:2454] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cfdc130877418dbe9d980d368973be in term 1.
I20240503 14:13:42.408769 31266 leader_election.cc:302] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82cfdc130877418dbe9d980d368973be, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:42.409009 32556 raft_consensus.cc:2781] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:42.409174 32556 raft_consensus.cc:686] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Becoming Leader. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Running, Role: LEADER
I20240503 14:13:42.409435 32556 consensus_queue.cc:237] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }
I20240503 14:13:42.412442 30796 catalog_manager.cc:5484] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be reported cstate change: term changed from 0 to 1, leader changed from <none> to 82cfdc130877418dbe9d980d368973be (127.22.242.68). New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:42.779907 32596 consensus_queue.cc:1035] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.787307 32596 consensus_queue.cc:1035] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:42.796002 32633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:42.796504 32633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:42.805624 32633 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 14:13:42.839131 32096 consensus_queue.cc:1035] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.844061 32096 consensus_queue.cc:1035] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.962921 31072 raft_consensus.cc:1261] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:42.963644 32556 consensus_queue.cc:1035] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.964982 31202 raft_consensus.cc:1261] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:42.972426 32556 consensus_queue.cc:1035] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
W20240503 14:13:43.115258 31135 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.65:46013, user_credentials={real_user=slave}} blocked reactor thread for 67923.7us
INSERT report
rows total: 2000
time total: 286.781 ms
time per row: 0.143391 ms
I20240503 14:13:43.257112 30794 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36072:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:43.266403 31311 tablet_service.cc:1467] Processing CreateTablet for tablet f17feb9b64214eaebcbe31dc45c75756 (DEFAULT_TABLE table=rebalance_test_table_9 [id=acc4ce6b628f4f498f830c4572e30d65]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.266566 30920 tablet_service.cc:1467] Processing CreateTablet for tablet f17feb9b64214eaebcbe31dc45c75756 (DEFAULT_TABLE table=rebalance_test_table_9 [id=acc4ce6b628f4f498f830c4572e30d65]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.266865 31311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f17feb9b64214eaebcbe31dc45c75756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.266958 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f17feb9b64214eaebcbe31dc45c75756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.266844 31180 tablet_service.cc:1467] Processing CreateTablet for tablet f17feb9b64214eaebcbe31dc45c75756 (DEFAULT_TABLE table=rebalance_test_table_9 [id=acc4ce6b628f4f498f830c4572e30d65]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.267272 31180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f17feb9b64214eaebcbe31dc45c75756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.272373 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 4ff2fa69f79644c5b93beeba32fc9ca0 (DEFAULT_TABLE table=rebalance_test_table_9 [id=acc4ce6b628f4f498f830c4572e30d65]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.272926 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff2fa69f79644c5b93beeba32fc9ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.277577 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 43221a82a1124bf8840c6369f27f47f2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=acc4ce6b628f4f498f830c4572e30d65]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.277992 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43221a82a1124bf8840c6369f27f47f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.277946 31310 tablet_service.cc:1467] Processing CreateTablet for tablet 43221a82a1124bf8840c6369f27f47f2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=acc4ce6b628f4f498f830c4572e30d65]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.278333 31310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43221a82a1124bf8840c6369f27f47f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.279852 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 43221a82a1124bf8840c6369f27f47f2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=acc4ce6b628f4f498f830c4572e30d65]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.280246 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43221a82a1124bf8840c6369f27f47f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.284550 31312 tablet_service.cc:1467] Processing CreateTablet for tablet 4ff2fa69f79644c5b93beeba32fc9ca0 (DEFAULT_TABLE table=rebalance_test_table_9 [id=acc4ce6b628f4f498f830c4572e30d65]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.284786 30921 tablet_service.cc:1467] Processing CreateTablet for tablet 4ff2fa69f79644c5b93beeba32fc9ca0 (DEFAULT_TABLE table=rebalance_test_table_9 [id=acc4ce6b628f4f498f830c4572e30d65]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.284925 31312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff2fa69f79644c5b93beeba32fc9ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.285126 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff2fa69f79644c5b93beeba32fc9ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.286334 32705 tablet_bootstrap.cc:492] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:43.288792 32705 tablet_bootstrap.cc:654] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.295300 32706 tablet_bootstrap.cc:492] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5: Bootstrap starting.
I20240503 14:13:43.295965 32703 tablet_bootstrap.cc:492] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:43.296830 32705 tablet_bootstrap.cc:492] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:43.297075 32705 ts_tablet_manager.cc:1367] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20240503 14:13:43.298007 32705 raft_consensus.cc:348] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:43.298398 32705 raft_consensus.cc:374] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.298539 32705 raft_consensus.cc:729] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.299044 32706 tablet_bootstrap.cc:654] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.298895 32705 consensus_queue.cc:260] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:43.299587 32705 ts_tablet_manager.cc:1398] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:43.299942 32705 tablet_bootstrap.cc:492] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:43.300465 32703 tablet_bootstrap.cc:654] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.301960 32705 tablet_bootstrap.cc:654] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.303335 32704 tablet_bootstrap.cc:492] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:43.306797 32703 tablet_bootstrap.cc:492] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:43.307010 32703 ts_tablet_manager.cc:1367] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20240503 14:13:43.307906 32703 raft_consensus.cc:348] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:43.308265 32703 raft_consensus.cc:374] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.308416 32703 raft_consensus.cc:729] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.308732 32703 consensus_queue.cc:260] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:43.309434 32703 ts_tablet_manager.cc:1398] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:43.309763 32703 tablet_bootstrap.cc:492] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Bootstrap starting.
I20240503 14:13:43.311399 32703 tablet_bootstrap.cc:654] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.313719 32706 tablet_bootstrap.cc:492] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5: No bootstrap required, opened a new log
I20240503 14:13:43.314852 32706 ts_tablet_manager.cc:1367] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.012s
I20240503 14:13:43.315222 32704 tablet_bootstrap.cc:654] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.316275 32706 raft_consensus.cc:348] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:43.316452 32706 raft_consensus.cc:374] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.316542 32706 raft_consensus.cc:729] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.316788 32706 consensus_queue.cc:260] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:43.317497 32706 ts_tablet_manager.cc:1398] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:43.317536 32705 tablet_bootstrap.cc:492] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:43.317821 32705 ts_tablet_manager.cc:1367] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.010s
I20240503 14:13:43.318928 32705 raft_consensus.cc:348] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:43.319109 32697 raft_consensus.cc:482] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:43.319308 32705 raft_consensus.cc:374] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.319257 32697 raft_consensus.cc:504] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:43.319424 32705 raft_consensus.cc:729] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.319754 32705 consensus_queue.cc:260] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:43.320008 32697 leader_election.cc:288] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:43.320508 32705 ts_tablet_manager.cc:1398] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:43.320796 32705 tablet_bootstrap.cc:492] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811: Bootstrap starting.
I20240503 14:13:43.320757 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17feb9b64214eaebcbe31dc45c75756" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be" is_pre_election: true
I20240503 14:13:43.321333 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17feb9b64214eaebcbe31dc45c75756" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:43.321585 30941 raft_consensus.cc:2454] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 0.
I20240503 14:13:43.322819 31135 leader_election.cc:302] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:43.323015 32703 tablet_bootstrap.cc:492] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce: No bootstrap required, opened a new log
I20240503 14:13:43.323173 32703 ts_tablet_manager.cc:1367] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.001s
I20240503 14:13:43.323354 32705 tablet_bootstrap.cc:654] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811: Neither blocks nor log segments found. Creating new log.
W20240503 14:13:43.323601 31137 leader_election.cc:341] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:43.324038 32703 raft_consensus.cc:348] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:43.324281 32703 raft_consensus.cc:374] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.325304 32703 raft_consensus.cc:729] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.325627 32703 consensus_queue.cc:260] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:43.326336 32703 ts_tablet_manager.cc:1398] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:43.324487 32697 raft_consensus.cc:2781] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:43.329035 32697 raft_consensus.cc:482] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:43.329128 32697 raft_consensus.cc:3037] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.332132 32697 raft_consensus.cc:504] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:43.333087 32704 tablet_bootstrap.cc:492] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:43.333266 32704 ts_tablet_manager.cc:1367] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.003s
I20240503 14:13:43.333500 32697 leader_election.cc:288] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 election: Requested vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:43.333722 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17feb9b64214eaebcbe31dc45c75756" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be"
I20240503 14:13:43.334129 32704 raft_consensus.cc:348] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
W20240503 14:13:43.334259 31137 leader_election.cc:341] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:43.334357 32704 raft_consensus.cc:374] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.334447 32704 raft_consensus.cc:729] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.334666 32704 consensus_queue.cc:260] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:43.335173 32704 ts_tablet_manager.cc:1398] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 14:13:43.335446 32704 tablet_bootstrap.cc:492] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:43.337733 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17feb9b64214eaebcbe31dc45c75756" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:43.337949 30941 raft_consensus.cc:3037] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.338044 32704 tablet_bootstrap.cc:654] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.340281 30941 raft_consensus.cc:2454] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 1.
I20240503 14:13:43.340745 31135 leader_election.cc:302] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters: 82cfdc130877418dbe9d980d368973be
I20240503 14:13:43.341082 32697 raft_consensus.cc:2781] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:43.341223 32697 raft_consensus.cc:686] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Becoming Leader. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Running, Role: LEADER
I20240503 14:13:43.341370 32704 tablet_bootstrap.cc:492] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:43.341444 32705 tablet_bootstrap.cc:492] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811: No bootstrap required, opened a new log
I20240503 14:13:43.341485 32704 ts_tablet_manager.cc:1367] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:13:43.341603 32705 ts_tablet_manager.cc:1367] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.009s
I20240503 14:13:43.341481 32697 consensus_queue.cc:237] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:43.342244 32704 raft_consensus.cc:348] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:43.342497 32704 raft_consensus.cc:374] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.342365 32705 raft_consensus.cc:348] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:43.342581 32704 raft_consensus.cc:729] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.342613 32705 raft_consensus.cc:374] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.342729 32705 raft_consensus.cc:729] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc902d91ccc435a82a5ede4a7729811, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.342815 32704 consensus_queue.cc:260] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:43.343008 32705 consensus_queue.cc:260] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:43.343405 32704 ts_tablet_manager.cc:1398] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:43.343816 32704 tablet_bootstrap.cc:492] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be: Bootstrap starting.
I20240503 14:13:43.344807 32705 ts_tablet_manager.cc:1398] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:43.346215 32704 tablet_bootstrap.cc:654] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.353374 32704 tablet_bootstrap.cc:492] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be: No bootstrap required, opened a new log
I20240503 14:13:43.353518 32704 ts_tablet_manager.cc:1367] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240503 14:13:43.354226 32704 raft_consensus.cc:348] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:43.354430 32704 raft_consensus.cc:374] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.354530 32704 raft_consensus.cc:729] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cfdc130877418dbe9d980d368973be, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.354730 32704 consensus_queue.cc:260] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:43.355226 32704 ts_tablet_manager.cc:1398] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:43.355185 30791 catalog_manager.cc:5484] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: term changed from 0 to 1, leader changed from <none> to e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67). New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:43.371207 32697 raft_consensus.cc:482] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:43.371389 32697 raft_consensus.cc:504] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:43.372025 32697 leader_election.cc:288] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:43.372522 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43221a82a1124bf8840c6369f27f47f2" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:43.372538 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43221a82a1124bf8840c6369f27f47f2" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be" is_pre_election: true
I20240503 14:13:43.372727 30941 raft_consensus.cc:2454] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 0.
I20240503 14:13:43.372730 31331 raft_consensus.cc:2454] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 0.
I20240503 14:13:43.373143 31135 leader_election.cc:302] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:43.373474 32697 raft_consensus.cc:2781] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:43.373585 32697 raft_consensus.cc:482] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:43.373703 32697 raft_consensus.cc:3037] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.373886 32691 raft_consensus.cc:482] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:43.374065 32691 raft_consensus.cc:504] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:43.374861 32691 leader_election.cc:288] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:43.375618 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff2fa69f79644c5b93beeba32fc9ca0" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:43.375741 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff2fa69f79644c5b93beeba32fc9ca0" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be" is_pre_election: true
I20240503 14:13:43.375820 30941 raft_consensus.cc:2454] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:43.375972 31331 raft_consensus.cc:2454] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 0.
I20240503 14:13:43.376232 31005 leader_election.cc:302] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 6dc4948ed7c740ab89b9e349d151b0e5; no voters:
I20240503 14:13:43.376395 32697 raft_consensus.cc:504] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:43.376521 32691 raft_consensus.cc:2781] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:43.376628 32691 raft_consensus.cc:482] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:43.376718 32691 raft_consensus.cc:3037] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.377085 32697 leader_election.cc:288] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 election: Requested vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:43.377348 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43221a82a1124bf8840c6369f27f47f2" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be"
I20240503 14:13:43.377507 31331 raft_consensus.cc:3037] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.378764 32691 raft_consensus.cc:504] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:43.379272 32691 leader_election.cc:288] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Requested vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:43.379578 31331 raft_consensus.cc:2454] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 1.
I20240503 14:13:43.379729 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff2fa69f79644c5b93beeba32fc9ca0" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:43.379906 30941 raft_consensus.cc:3037] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.380000 31137 leader_election.cc:302] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82cfdc130877418dbe9d980d368973be, e7badbe63a8a4d9ca0c78818c6f4c8ce; no voters:
I20240503 14:13:43.380059 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff2fa69f79644c5b93beeba32fc9ca0" candidate_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82cfdc130877418dbe9d980d368973be"
I20240503 14:13:43.380259 31331 raft_consensus.cc:3037] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.380353 32697 raft_consensus.cc:2781] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:43.380523 32697 raft_consensus.cc:686] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Becoming Leader. State: Replica: e7badbe63a8a4d9ca0c78818c6f4c8ce, State: Running, Role: LEADER
I20240503 14:13:43.380796 32697 consensus_queue.cc:237] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:43.382370 30941 raft_consensus.cc:2454] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:43.382676 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43221a82a1124bf8840c6369f27f47f2" candidate_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811"
I20240503 14:13:43.382737 31005 leader_election.cc:302] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc902d91ccc435a82a5ede4a7729811, 6dc4948ed7c740ab89b9e349d151b0e5; no voters:
I20240503 14:13:43.382849 30940 raft_consensus.cc:3037] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.383347 31331 raft_consensus.cc:2454] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc4948ed7c740ab89b9e349d151b0e5 in term 1.
I20240503 14:13:43.384302 32556 raft_consensus.cc:482] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:43.384434 32556 raft_consensus.cc:504] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }
I20240503 14:13:43.384332 30796 catalog_manager.cc:5484] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: term changed from 0 to 1, leader changed from <none> to e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67). New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:43.385061 32556 leader_election.cc:288] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:43.385284 30940 raft_consensus.cc:2454] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in term 1.
I20240503 14:13:43.385602 32691 raft_consensus.cc:2781] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:43.385799 31200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17feb9b64214eaebcbe31dc45c75756" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" is_pre_election: true
I20240503 14:13:43.385861 32691 raft_consensus.cc:686] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Becoming Leader. State: Replica: 6dc4948ed7c740ab89b9e349d151b0e5, State: Running, Role: LEADER
I20240503 14:13:43.386138 32691 consensus_queue.cc:237] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } }
I20240503 14:13:43.386312 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17feb9b64214eaebcbe31dc45c75756" candidate_uuid: "82cfdc130877418dbe9d980d368973be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc902d91ccc435a82a5ede4a7729811" is_pre_election: true
I20240503 14:13:43.386617 30941 raft_consensus.cc:2379] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82cfdc130877418dbe9d980d368973be in current term 1: Already voted for candidate e7badbe63a8a4d9ca0c78818c6f4c8ce in this term.
I20240503 14:13:43.387907 31266 leader_election.cc:302] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82cfdc130877418dbe9d980d368973be; no voters: 0dc902d91ccc435a82a5ede4a7729811, e7badbe63a8a4d9ca0c78818c6f4c8ce
I20240503 14:13:43.388360 32556 raft_consensus.cc:3037] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.390236 32556 raft_consensus.cc:2726] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:43.392032 30796 catalog_manager.cc:5484] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:43.757272 32697 consensus_queue.cc:1035] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:43.765727 32719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:43.766170 32719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:13:43.767403 32694 consensus_queue.cc:1035] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.772083 32694 consensus_queue.cc:1035] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:43.781723 32719 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:43.790668 32697 consensus_queue.cc:1035] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240503 14:13:43.921159 32691 consensus_queue.cc:1035] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.927156 31332 raft_consensus.cc:1261] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:43.927899 32691 consensus_queue.cc:1035] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 263.176 ms
time per row: 0.131588 ms
I20240503 14:13:44.219044 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.69:42189
--local_ip_for_outbound_sockets=127.22.242.69
--tserver_master_addrs=127.22.242.126:34247
--webserver_port=44473
--webserver_interface=127.22.242.69
--builtin_ntp_servers=127.22.242.84:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:13:44.561697 314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:44.562072 314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:44.562218 314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:44.570611 314 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:44.570756 314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:44.570943 314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.69
I20240503 14:13:44.581698 314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.69:42189
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.242.69
--webserver_port=44473
--tserver_master_addrs=127.22.242.126:34247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.69
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:44.583132 314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:44.585047 314 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:44.598994 320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:44.598994 319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:44.605650 322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:44.606109 314 server_base.cc:1034] running on GCE node
I20240503 14:13:44.606865 314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:44.608152 314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:44.609391 314 hybrid_clock.cc:648] HybridClock initialized: now 1714745624609322 us; error 62 us; skew 500 ppm
I20240503 14:13:44.609762 314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:44.612476 314 webserver.cc:460] Webserver started at http://127.22.242.69:44473/ using document root <none> and password file <none>
I20240503 14:13:44.613411 314 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:44.613559 314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:44.620079 314 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:13:44.623188 328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:44.624652 314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:44.624812 314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6967bdda50404d68a5ac043b709440e1"
format_stamp: "Formatted at 2024-05-03 14:13:23 on dist-test-slave-r3pd"
I20240503 14:13:44.625861 314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:44.654706 314 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:44.655557 314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:44.655818 314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:44.656944 314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:44.658540 314 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:44.658612 314 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:44.658699 314 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:44.658742 314 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:44.695665 314 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.69:42189
I20240503 14:13:44.695683 440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.69:42189 every 8 connection(s)
I20240503 14:13:44.697539 314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:13:44.699128 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 314
I20240503 14:13:44.729933 441 heartbeater.cc:347] Connected to a master server at 127.22.242.126:34247
I20240503 14:13:44.730304 441 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:44.731279 441 heartbeater.cc:511] Master 127.22.242.126:34247 requested a full tablet report, sending...
I20240503 14:13:44.733042 30797 ts_manager.cc:194] Re-registered known tserver with Master: 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189)
I20240503 14:13:44.734283 30797 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.69:58185
I20240503 14:13:44.699882 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.70:46829
--local_ip_for_outbound_sockets=127.22.242.70
--tserver_master_addrs=127.22.242.126:34247
--webserver_port=45503
--webserver_interface=127.22.242.70
--builtin_ntp_servers=127.22.242.84:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:13:45.092790 445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:45.093101 445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:45.093201 445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:45.101394 445 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:45.101537 445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:45.101686 445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.70
I20240503 14:13:45.110721 445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.70:46829
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.242.70
--webserver_port=45503
--tserver_master_addrs=127.22.242.126:34247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.70
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:45.112046 445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:45.113739 445 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:45.125659 451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:45.125648 452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:45.125937 454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:45.126451 445 server_base.cc:1034] running on GCE node
I20240503 14:13:45.127034 445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:45.128001 445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:45.129202 445 hybrid_clock.cc:648] HybridClock initialized: now 1714745625129143 us; error 55 us; skew 500 ppm
I20240503 14:13:45.129559 445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:45.131800 445 webserver.cc:460] Webserver started at http://127.22.242.70:45503/ using document root <none> and password file <none>
I20240503 14:13:45.132555 445 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:45.132676 445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:45.139237 445 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.003s
I20240503 14:13:45.142987 460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:45.144551 445 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:45.144721 445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "fa71cbdf0e544b81bd4e7e27196d2c91"
format_stamp: "Formatted at 2024-05-03 14:13:24 on dist-test-slave-r3pd"
I20240503 14:13:45.145757 445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:45.170158 445 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:45.170938 445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:45.171196 445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:45.172118 445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:45.173708 445 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:45.173784 445 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:45.173907 445 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:45.173956 445 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:45.213582 445 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.70:46829
I20240503 14:13:45.213631 572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.70:46829 every 8 connection(s)
I20240503 14:13:45.215121 445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:13:45.224519 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 445
I20240503 14:13:45.225255 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.71:46453
--local_ip_for_outbound_sockets=127.22.242.71
--tserver_master_addrs=127.22.242.126:34247
--webserver_port=43355
--webserver_interface=127.22.242.71
--builtin_ntp_servers=127.22.242.84:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:13:45.228024 573 heartbeater.cc:347] Connected to a master server at 127.22.242.126:34247
I20240503 14:13:45.228349 573 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:45.229347 573 heartbeater.cc:511] Master 127.22.242.126:34247 requested a full tablet report, sending...
I20240503 14:13:45.230832 30797 ts_manager.cc:194] Re-registered known tserver with Master: fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829)
I20240503 14:13:45.231751 30797 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.70:59257
W20240503 14:13:45.575222 576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:45.575531 576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:45.575682 576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:45.583670 576 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:45.583842 576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:45.584004 576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.71
I20240503 14:13:45.596064 576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.71:46453
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.242.71
--webserver_port=43355
--tserver_master_addrs=127.22.242.126:34247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.71
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:45.597668 576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:45.599848 576 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:45.613068 583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:45.613188 582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:45.613013 585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:45.613967 576 server_base.cc:1034] running on GCE node
I20240503 14:13:45.614565 576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:45.615473 576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:45.616690 576 hybrid_clock.cc:648] HybridClock initialized: now 1714745625616636 us; error 47 us; skew 500 ppm
I20240503 14:13:45.617023 576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:45.619367 576 webserver.cc:460] Webserver started at http://127.22.242.71:43355/ using document root <none> and password file <none>
I20240503 14:13:45.620249 576 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:45.620373 576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:45.625510 576 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:13:45.628096 591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:45.629506 576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:45.629683 576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4665a3377574417ba0d3165c236533c0"
format_stamp: "Formatted at 2024-05-03 14:13:24 on dist-test-slave-r3pd"
I20240503 14:13:45.630745 576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:45.649708 576 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:45.650471 576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:45.650727 576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:45.651674 576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:45.653276 576 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:45.653347 576 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:45.653443 576 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:45.653489 576 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:45.705132 576 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.71:46453
I20240503 14:13:45.705199 704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.71:46453 every 8 connection(s)
I20240503 14:13:45.706861 576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:13:45.716619 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 576
I20240503 14:13:45.717245 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.72:36545
--local_ip_for_outbound_sockets=127.22.242.72
--tserver_master_addrs=127.22.242.126:34247
--webserver_port=44281
--webserver_interface=127.22.242.72
--builtin_ntp_servers=127.22.242.84:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:13:45.722481 705 heartbeater.cc:347] Connected to a master server at 127.22.242.126:34247
I20240503 14:13:45.723047 705 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:45.723836 705 heartbeater.cc:511] Master 127.22.242.126:34247 requested a full tablet report, sending...
I20240503 14:13:45.725229 30797 ts_manager.cc:194] Re-registered known tserver with Master: 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453)
I20240503 14:13:45.725952 30797 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.71:49117
I20240503 14:13:45.736884 441 heartbeater.cc:503] Master 127.22.242.126:34247 was elected leader, sending a full tablet report...
W20240503 14:13:46.065526 708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:46.065837 708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:46.065976 708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:46.073824 708 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:46.073997 708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:46.074116 708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.72
I20240503 14:13:46.083148 708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.72:36545
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.242.72
--webserver_port=44281
--tserver_master_addrs=127.22.242.126:34247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.72
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:46.084448 708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:46.086221 708 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:46.099834 716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:46.099795 715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:46.099810 718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:46.100800 708 server_base.cc:1034] running on GCE node
I20240503 14:13:46.101446 708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:46.102535 708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:46.103754 708 hybrid_clock.cc:648] HybridClock initialized: now 1714745626103710 us; error 38 us; skew 500 ppm
I20240503 14:13:46.104085 708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:46.106611 708 webserver.cc:460] Webserver started at http://127.22.242.72:44281/ using document root <none> and password file <none>
I20240503 14:13:46.107287 708 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:46.107395 708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:46.112543 708 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:13:46.115182 724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:46.116499 708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 14:13:46.116640 708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "85f69eefe4294197923dff5efadcbbd5"
format_stamp: "Formatted at 2024-05-03 14:13:25 on dist-test-slave-r3pd"
I20240503 14:13:46.117401 708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:46.133509 708 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:46.134303 708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:46.134573 708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:46.135614 708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:46.137436 708 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:46.137516 708 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:46.137639 708 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:46.137694 708 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:46.176573 708 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.72:36545
I20240503 14:13:46.176652 836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.72:36545 every 8 connection(s)
I20240503 14:13:46.177981 708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:13:46.187750 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 708
I20240503 14:13:46.191269 837 heartbeater.cc:347] Connected to a master server at 127.22.242.126:34247
I20240503 14:13:46.191598 837 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:46.192695 837 heartbeater.cc:511] Master 127.22.242.126:34247 requested a full tablet report, sending...
I20240503 14:13:46.194279 30797 ts_manager.cc:194] Re-registered known tserver with Master: 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545)
I20240503 14:13:46.195399 30797 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.72:60221
I20240503 14:13:46.234992 573 heartbeater.cc:503] Master 127.22.242.126:34247 was elected leader, sending a full tablet report...
I20240503 14:13:46.728368 705 heartbeater.cc:503] Master 127.22.242.126:34247 was elected leader, sending a full tablet report...
I20240503 14:13:47.198053 837 heartbeater.cc:503] Master 127.22.242.126:34247 was elected leader, sending a full tablet report...
W20240503 14:13:47.555996 31138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.66:33221, user_credentials={real_user=slave}} blocked reactor thread for 55936.4us
I20240503 14:13:47.754096 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:47.824859 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:47.851939 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:47.855732 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:47.863433 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:47.868588 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:47.879107 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:47.933964 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:47.964797 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:47.985689 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:47.996688 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:48.022545 508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:48.025682 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:48.036075 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:48.069751 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:48.084688 508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:48.120239 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:48.122664 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:48.125963 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:48.129601 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:48.149466 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:48.150987 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:48.159265 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:48.165953 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:48.190021 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:48.232494 508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:48.258288 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:48.280895 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:48.312224 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:48.331357 508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:48.373790 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:48.382834 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:48.517453 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:48.560650 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:48.615957 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:13:48.646168 30876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.68:37063, user_credentials={real_user=slave}} blocked reactor thread for 77587.7us
I20240503 14:13:48.652704 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:48.684299 508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:48.686655 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:48.697688 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:48.700241 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:49.441614 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:49.475106 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:49.487154 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:49.518777 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:49.520953 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:49.533399 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:49.562214 508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:49.563876 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:49.598898 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:49.613705 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:49.614800 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:49.658492 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:49.677861 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:49.717347 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:49.720630 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:49.726083 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:49.770851 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:49.783655 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:49.790829 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:49.801321 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:49.803416 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:49.829488 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:49.834431 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:49.859228 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:49.874902 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:49.879310 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:49.889389 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:49.891172 508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:49.909658 508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:49.966507 508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:49.976321 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:49.978219 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:50.201138 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:50.210386 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:50.240762 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:50.268986 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:50.286257 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:50.298383 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:50.299947 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:50.354759 508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.019182 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:51.051170 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:51.061897 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:13:51.126897 31006 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.67:44297, user_credentials={real_user=slave}} blocked reactor thread for 47973.5us
W20240503 14:13:51.158025 31005 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.65:46013, user_credentials={real_user=slave}} blocked reactor thread for 78678.1us
I20240503 14:13:51.166142 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.169376 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:51.186410 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:51.191964 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:51.196928 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.218789 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:51.223551 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.238013 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.265785 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.270383 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:51.316959 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:51.318723 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.319032 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.327839 508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.352128 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:51.353574 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:51.354493 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:51.367583 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.390228 508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.405705 508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.462250 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:51.541666 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:51.591437 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.595052 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.610399 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:51.643622 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.671021 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:51.714545 508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.772758 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:51.781569 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:51.819244 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.832574 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.843858 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:51.869050 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:51.871105 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:51.872084 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:51.883212 508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:52.017774 31071 consensus_queue.cc:237] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:52.020229 31201 consensus_queue.cc:237] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:52.036615 31331 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.037539 1387 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:52.047029 30938 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.047658 1387 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:52.054550 1387 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:52.056571 30938 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:52.058912 31332 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.059556 1600 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:52.061077 30939 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:52.060549 31138 consensus_peers.cc:485] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f82073122a74268a464387659dd90bf. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:52.061676 1600 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:52.072078 30792 catalog_manager.cc:5484] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:52.089602 31331 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:52.099676 31201 consensus_queue.cc:237] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:52.104939 30938 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:13:52.105705 31138 consensus_peers.cc:485] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f82073122a74268a464387659dd90bf. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:52.106240 31331 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:52.106884 1387 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:52.107693 1387 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:52.111953 1575 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
W20240503 14:13:52.115926 31138 consensus_peers.cc:485] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189): Couldn't send request to peer 6967bdda50404d68a5ac043b709440e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f82073122a74268a464387659dd90bf. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:52.117609 31331 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:52.118044 30941 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:52.127456 1537 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:52.128952 30941 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:52.129544 31331 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:52.140828 30797 catalog_manager.cc:5484] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:13:52.173424 31008 consensus_peers.cc:485] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63b2120788a54af78f02449f46bf23c7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:52.196060 30794 catalog_manager.cc:5484] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index 4 to 5, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:52.223625 31201 consensus_queue.cc:237] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:52.224793 31071 consensus_queue.cc:237] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
W20240503 14:13:52.238399 31138 consensus_peers.cc:485] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189): Couldn't send request to peer 6967bdda50404d68a5ac043b709440e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1bb75d7d9499475ca60c633c0801d639. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:52.238622 31072 raft_consensus.cc:1261] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.240324 31331 raft_consensus.cc:1261] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.240832 31332 raft_consensus.cc:1261] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.241433 1537 consensus_queue.cc:1035] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:52.246093 1631 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:52.248310 31201 raft_consensus.cc:1261] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.249250 1537 consensus_queue.cc:1035] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:52.250245 1631 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:13:52.256238 31005 consensus_peers.cc:485] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829): Couldn't send request to peer fa71cbdf0e544b81bd4e7e27196d2c91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c837def6d76242f08b20a448541e64d6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:52.270097 1537 raft_consensus.cc:2932] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:52.273119 31200 raft_consensus.cc:2932] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:52.275426 31330 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:52.274596 1631 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:52.280683 30794 catalog_manager.cc:5484] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:52.286223 31071 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:52.288404 30792 catalog_manager.cc:5484] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:52.323264 31332 raft_consensus.cc:2932] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:52.361135 31072 consensus_queue.cc:237] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:52.373489 31202 raft_consensus.cc:1261] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:52.374028 31005 consensus_peers.cc:485] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829): Couldn't send request to peer fa71cbdf0e544b81bd4e7e27196d2c91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7b5dd5eb46a45989bf9ffe3144bc189. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:52.374244 31332 raft_consensus.cc:1261] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.374307 1600 consensus_queue.cc:1035] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:52.374856 1537 consensus_queue.cc:1035] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:52.383155 1600 raft_consensus.cc:2932] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:52.383805 31202 consensus_queue.cc:237] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:52.384380 31201 raft_consensus.cc:2932] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:52.388468 31200 consensus_queue.cc:237] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:52.390795 30794 catalog_manager.cc:5484] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:52.392807 31331 raft_consensus.cc:2932] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:52.395491 31332 raft_consensus.cc:1261] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:52.396450 1387 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:13:52.398895 31138 consensus_peers.cc:485] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189): Couldn't send request to peer 6967bdda50404d68a5ac043b709440e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1bb75d7d9499475ca60c633c0801d639. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:52.399158 31138 consensus_peers.cc:485] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1bb75d7d9499475ca60c633c0801d639. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:52.403177 31072 raft_consensus.cc:1261] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:52.404521 1631 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:52.407939 1631 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:52.414263 31071 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:52.421422 31332 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:52.425076 30796 catalog_manager.cc:5484] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 4 to 5, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:52.478715 31332 raft_consensus.cc:1261] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.479437 1631 consensus_queue.cc:1035] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:52.484347 30939 raft_consensus.cc:1261] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.485000 1631 consensus_queue.cc:1035] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:13:52.487569 31135 consensus_peers.cc:485] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829): Couldn't send request to peer fa71cbdf0e544b81bd4e7e27196d2c91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43221a82a1124bf8840c6369f27f47f2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:52.489992 1628 raft_consensus.cc:2932] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:52.490433 31332 raft_consensus.cc:2932] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:52.490664 31071 consensus_queue.cc:237] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:52.495625 30939 raft_consensus.cc:2932] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:52.500696 30792 catalog_manager.cc:5484] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:52.505532 30941 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:52.506104 1537 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:52.507295 31332 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:52.508433 1537 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:13:52.509842 31008 consensus_peers.cc:485] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63b2120788a54af78f02449f46bf23c7. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:52.511996 31008 consensus_peers.cc:485] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189): Couldn't send request to peer 6967bdda50404d68a5ac043b709440e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63b2120788a54af78f02449f46bf23c7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:52.513142 1602 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:52.518177 30941 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:52.521075 31332 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:52.525879 30794 catalog_manager.cc:5484] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 4 to 5, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:52.528077 31331 consensus_queue.cc:237] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:52.541023 31202 raft_consensus.cc:1261] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.541999 30941 raft_consensus.cc:1261] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.542505 1349 consensus_queue.cc:1035] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:52.558867 1349 consensus_queue.cc:1035] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20240503 14:13:52.566279 1722 raft_consensus.cc:2932] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:52.568789 31202 raft_consensus.cc:2932] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:52.569669 1737 ts_tablet_manager.cc:902] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1: Initiating tablet copy from peer e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:52.578534 31331 consensus_queue.cc:237] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:52.579157 30796 catalog_manager.cc:5484] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:52.608036 31202 raft_consensus.cc:1261] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.609557 1578 consensus_queue.cc:1035] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:52.611272 30939 raft_consensus.cc:1261] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.612908 1578 consensus_queue.cc:1035] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:13:52.619486 31268 consensus_peers.cc:485] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f45824b1fff4302abd03c1fb48c6c7b. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:52.619786 31268 consensus_peers.cc:485] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55cf34de74694f47ad3f9f4d9bdffae4. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:52.639591 30941 raft_consensus.cc:2932] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:52.651630 1722 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:52.657317 31200 consensus_queue.cc:237] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:52.657866 30938 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:52.663151 30797 catalog_manager.cc:5484] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:52.676424 31202 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:52.710897 1763 ts_tablet_manager.cc:902] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5: Initiating tablet copy from peer e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:52.713131 1737 tablet_copy_client.cc:308] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44297
I20240503 14:13:52.714052 31331 raft_consensus.cc:1261] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.714697 30941 raft_consensus.cc:1261] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.715538 1628 consensus_queue.cc:1035] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:52.715852 1631 consensus_queue.cc:1035] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:13:52.716580 31138 consensus_peers.cc:485] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189): Couldn't send request to peer 6967bdda50404d68a5ac043b709440e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f17feb9b64214eaebcbe31dc45c75756. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:52.738233 1575 raft_consensus.cc:2932] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:52.739101 30939 raft_consensus.cc:2932] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:52.744939 30797 catalog_manager.cc:5484] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:52.757256 31332 raft_consensus.cc:2932] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:52.780244 31222 tablet_copy_service.cc:140] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Received BeginTabletCopySession request for tablet 2f82073122a74268a464387659dd90bf from peer 6967bdda50404d68a5ac043b709440e1 ({username='slave'} at 127.22.242.69:38549)
I20240503 14:13:52.780581 31222 tablet_copy_service.cc:161] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Beginning new tablet copy session on tablet 2f82073122a74268a464387659dd90bf from peer 6967bdda50404d68a5ac043b709440e1 at {username='slave'} at 127.22.242.69:38549: session id = 6967bdda50404d68a5ac043b709440e1-2f82073122a74268a464387659dd90bf
I20240503 14:13:52.784730 31222 tablet_copy_source_session.cc:205] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:52.791157 1737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f82073122a74268a464387659dd90bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.797518 1787 ts_tablet_manager.cc:902] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1: Initiating tablet copy from peer e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:52.806012 1763 tablet_copy_client.cc:308] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44297
I20240503 14:13:52.823467 1799 ts_tablet_manager.cc:902] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:52.858883 1799 tablet_copy_client.cc:308] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:13:52.862859 1805 ts_tablet_manager.cc:902] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5: Initiating tablet copy from peer e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:52.862368 1804 ts_tablet_manager.cc:902] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:52.865226 31332 consensus_queue.cc:237] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:52.873606 31222 tablet_copy_service.cc:140] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Received BeginTabletCopySession request for tablet 2f82073122a74268a464387659dd90bf from peer 85f69eefe4294197923dff5efadcbbd5 ({username='slave'} at 127.22.242.72:40123)
I20240503 14:13:52.873808 31222 tablet_copy_service.cc:161] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Beginning new tablet copy session on tablet 2f82073122a74268a464387659dd90bf from peer 85f69eefe4294197923dff5efadcbbd5 at {username='slave'} at 127.22.242.72:40123: session id = 85f69eefe4294197923dff5efadcbbd5-2f82073122a74268a464387659dd90bf
I20240503 14:13:52.875831 31200 consensus_queue.cc:237] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:52.876847 31222 tablet_copy_source_session.cc:205] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:52.886056 1763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f82073122a74268a464387659dd90bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.892576 1787 tablet_copy_client.cc:308] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44297
I20240503 14:13:52.893745 31222 tablet_copy_service.cc:140] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Received BeginTabletCopySession request for tablet 1bb75d7d9499475ca60c633c0801d639 from peer 6967bdda50404d68a5ac043b709440e1 ({username='slave'} at 127.22.242.69:38549)
I20240503 14:13:52.893934 31222 tablet_copy_service.cc:161] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Beginning new tablet copy session on tablet 1bb75d7d9499475ca60c633c0801d639 from peer 6967bdda50404d68a5ac043b709440e1 at {username='slave'} at 127.22.242.69:38549: session id = 6967bdda50404d68a5ac043b709440e1-1bb75d7d9499475ca60c633c0801d639
I20240503 14:13:52.897925 31222 tablet_copy_source_session.cc:205] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:52.900038 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb75d7d9499475ca60c633c0801d639. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.901463 30941 raft_consensus.cc:1261] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.902151 30939 raft_consensus.cc:1261] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.902778 1578 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:52.902895 31071 raft_consensus.cc:1261] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.904011 1628 consensus_queue.cc:1035] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:52.905298 1578 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:52.926344 1822 ts_tablet_manager.cc:902] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
W20240503 14:13:52.931643 31135 consensus_peers.cc:485] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829): Couldn't send request to peer fa71cbdf0e544b81bd4e7e27196d2c91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91a7b8a53140429b956178d43478a541. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:52.947320 1827 ts_tablet_manager.cc:902] T 0f45824b1fff4302abd03c1fb48c6c7b P 4665a3377574417ba0d3165c236533c0: Initiating tablet copy from peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:52.956249 1818 ts_tablet_manager.cc:902] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:52.972921 31332 raft_consensus.cc:1261] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:52.973771 1628 consensus_queue.cc:1035] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:52.991815 1628 raft_consensus.cc:2932] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:52.993111 31332 raft_consensus.cc:2932] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:53.001037 30797 catalog_manager.cc:5484] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:53.007614 1722 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:53.012450 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:53.012755 31072 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:53.014427 30939 raft_consensus.cc:2932] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:53.015947 1827 tablet_copy_client.cc:308] T 0f45824b1fff4302abd03c1fb48c6c7b P 4665a3377574417ba0d3165c236533c0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.68:37063
I20240503 14:13:53.016458 1804 tablet_copy_client.cc:308] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:13:53.019063 1817 ts_tablet_manager.cc:902] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91: Initiating tablet copy from peer e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:53.024063 30794 catalog_manager.cc:5484] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:53.033533 1805 tablet_copy_client.cc:308] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44297
I20240503 14:13:53.035940 31222 tablet_copy_service.cc:140] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Received BeginTabletCopySession request for tablet 1bb75d7d9499475ca60c633c0801d639 from peer 85f69eefe4294197923dff5efadcbbd5 ({username='slave'} at 127.22.242.72:40123)
I20240503 14:13:53.036258 31222 tablet_copy_service.cc:161] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Beginning new tablet copy session on tablet 1bb75d7d9499475ca60c633c0801d639 from peer 85f69eefe4294197923dff5efadcbbd5 at {username='slave'} at 127.22.242.72:40123: session id = 85f69eefe4294197923dff5efadcbbd5-1bb75d7d9499475ca60c633c0801d639
I20240503 14:13:53.038522 30939 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:53.039249 31222 tablet_copy_source_session.cc:205] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:53.050773 1851 ts_tablet_manager.cc:902] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0: Initiating tablet copy from peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:53.065888 1805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb75d7d9499475ca60c633c0801d639. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:13:53.069067 31268 consensus_peers.cc:485] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 626fcb185af0446b89ae82066e08981d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:53.076664 1818 tablet_copy_client.cc:308] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:13:53.095423 1822 tablet_copy_client.cc:308] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:13:53.097050 1737 tablet_copy_client.cc:783] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:53.097576 1737 tablet_copy_client.cc:647] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:53.101043 1737 tablet_copy_client.cc:523] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:53.105327 1787 tablet_copy_client.cc:783] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:53.106051 31352 tablet_copy_service.cc:140] P 82cfdc130877418dbe9d980d368973be: Received BeginTabletCopySession request for tablet 0f45824b1fff4302abd03c1fb48c6c7b from peer 4665a3377574417ba0d3165c236533c0 ({username='slave'} at 127.22.242.71:44595)
I20240503 14:13:53.106344 1787 tablet_copy_client.cc:647] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:53.106319 31352 tablet_copy_service.cc:161] P 82cfdc130877418dbe9d980d368973be: Beginning new tablet copy session on tablet 0f45824b1fff4302abd03c1fb48c6c7b from peer 4665a3377574417ba0d3165c236533c0 at {username='slave'} at 127.22.242.71:44595: session id = 4665a3377574417ba0d3165c236533c0-0f45824b1fff4302abd03c1fb48c6c7b
I20240503 14:13:53.110422 31352 tablet_copy_source_session.cc:205] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:53.114941 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet 63b2120788a54af78f02449f46bf23c7 from peer 6967bdda50404d68a5ac043b709440e1 ({username='slave'} at 127.22.242.69:44021)
I20240503 14:13:53.115123 31089 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet 63b2120788a54af78f02449f46bf23c7 from peer 85f69eefe4294197923dff5efadcbbd5 ({username='slave'} at 127.22.242.72:54703)
I20240503 14:13:53.115260 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet 63b2120788a54af78f02449f46bf23c7 from peer 6967bdda50404d68a5ac043b709440e1 at {username='slave'} at 127.22.242.69:44021: session id = 6967bdda50404d68a5ac043b709440e1-63b2120788a54af78f02449f46bf23c7
I20240503 14:13:53.116364 31091 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet c837def6d76242f08b20a448541e64d6 from peer fa71cbdf0e544b81bd4e7e27196d2c91 ({username='slave'} at 127.22.242.70:40181)
I20240503 14:13:53.116597 31091 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet c837def6d76242f08b20a448541e64d6 from peer fa71cbdf0e544b81bd4e7e27196d2c91 at {username='slave'} at 127.22.242.70:40181: session id = fa71cbdf0e544b81bd4e7e27196d2c91-c837def6d76242f08b20a448541e64d6
I20240503 14:13:53.119663 1827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f45824b1fff4302abd03c1fb48c6c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.119686 31090 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet e7b5dd5eb46a45989bf9ffe3144bc189 from peer fa71cbdf0e544b81bd4e7e27196d2c91 ({username='slave'} at 127.22.242.70:40181)
I20240503 14:13:53.119879 31090 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet e7b5dd5eb46a45989bf9ffe3144bc189 from peer fa71cbdf0e544b81bd4e7e27196d2c91 at {username='slave'} at 127.22.242.70:40181: session id = fa71cbdf0e544b81bd4e7e27196d2c91-e7b5dd5eb46a45989bf9ffe3144bc189
I20240503 14:13:53.123126 31090 tablet_copy_source_session.cc:205] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:53.124861 31089 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet 63b2120788a54af78f02449f46bf23c7 from peer 85f69eefe4294197923dff5efadcbbd5 at {username='slave'} at 127.22.242.72:54703: session id = 85f69eefe4294197923dff5efadcbbd5-63b2120788a54af78f02449f46bf23c7
I20240503 14:13:53.125929 1787 tablet_copy_client.cc:523] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:53.129325 31089 tablet_copy_source_session.cc:205] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:53.129943 31091 tablet_copy_source_session.cc:205] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:53.130390 31092 tablet_copy_source_session.cc:205] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:53.131615 1804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b5dd5eb46a45989bf9ffe3144bc189. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.135259 1799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c837def6d76242f08b20a448541e64d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.140457 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:53.141316 1822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b2120788a54af78f02449f46bf23c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.146281 1818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b2120788a54af78f02449f46bf23c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.162981 1787 tablet_bootstrap.cc:492] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1: Bootstrap starting.
I20240503 14:13:53.178988 1763 tablet_copy_client.cc:783] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:53.180183 1763 tablet_copy_client.cc:647] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:53.182744 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:53.183763 1805 tablet_copy_client.cc:783] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:53.196529 1805 tablet_copy_client.cc:647] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:53.197821 1804 tablet_copy_client.cc:783] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:53.198204 1804 tablet_copy_client.cc:647] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:53.170445 1818 tablet_copy_client.cc:783] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:53.198511 1818 tablet_copy_client.cc:647] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:53.178537 1737 tablet_bootstrap.cc:492] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1: Bootstrap starting.
I20240503 14:13:53.204346 1763 tablet_copy_client.cc:523] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:53.207084 1805 tablet_copy_client.cc:523] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:53.208668 1882 ts_tablet_manager.cc:902] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1: Initiating tablet copy from peer e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:53.210570 1817 tablet_copy_client.cc:308] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44297
I20240503 14:13:53.213142 1851 tablet_copy_client.cc:308] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.68:37063
I20240503 14:13:53.214272 31352 tablet_copy_service.cc:140] P 82cfdc130877418dbe9d980d368973be: Received BeginTabletCopySession request for tablet 55cf34de74694f47ad3f9f4d9bdffae4 from peer 4665a3377574417ba0d3165c236533c0 ({username='slave'} at 127.22.242.71:44595)
I20240503 14:13:53.214494 31352 tablet_copy_service.cc:161] P 82cfdc130877418dbe9d980d368973be: Beginning new tablet copy session on tablet 55cf34de74694f47ad3f9f4d9bdffae4 from peer 4665a3377574417ba0d3165c236533c0 at {username='slave'} at 127.22.242.71:44595: session id = 4665a3377574417ba0d3165c236533c0-55cf34de74694f47ad3f9f4d9bdffae4
I20240503 14:13:53.217691 31352 tablet_copy_source_session.cc:205] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:53.219235 1851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55cf34de74694f47ad3f9f4d9bdffae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.219844 31221 tablet_copy_service.cc:140] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Received BeginTabletCopySession request for tablet 43221a82a1124bf8840c6369f27f47f2 from peer fa71cbdf0e544b81bd4e7e27196d2c91 ({username='slave'} at 127.22.242.70:33209)
I20240503 14:13:53.220005 31221 tablet_copy_service.cc:161] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Beginning new tablet copy session on tablet 43221a82a1124bf8840c6369f27f47f2 from peer fa71cbdf0e544b81bd4e7e27196d2c91 at {username='slave'} at 127.22.242.70:33209: session id = fa71cbdf0e544b81bd4e7e27196d2c91-43221a82a1124bf8840c6369f27f47f2
I20240503 14:13:53.223343 31221 tablet_copy_source_session.cc:205] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:53.223752 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:53.226852 1804 tablet_copy_client.cc:523] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:53.228677 1818 tablet_copy_client.cc:523] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:53.232360 1822 tablet_copy_client.cc:783] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:53.232699 1822 tablet_copy_client.cc:647] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:53.246937 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:53.247566 1822 tablet_copy_client.cc:523] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:53.255100 1817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43221a82a1124bf8840c6369f27f47f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.260241 1799 tablet_copy_client.cc:783] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:53.260557 1799 tablet_copy_client.cc:647] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:53.279690 1799 tablet_copy_client.cc:523] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:53.287833 1818 tablet_bootstrap.cc:492] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1: Bootstrap starting.
I20240503 14:13:53.287739 31331 consensus_queue.cc:237] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:53.293515 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:53.295054 30939 consensus_queue.cc:237] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:53.305972 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:53.306998 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:53.310261 1817 tablet_copy_client.cc:783] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 0 data blocks...
W20240503 14:13:53.313417 31268 consensus_peers.cc:485] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69b741aad35d4bfc8fea5b8e21d4a733. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:53.314195 31202 raft_consensus.cc:1261] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:53.314538 31200 raft_consensus.cc:1261] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:53.315001 1850 consensus_queue.cc:1035] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:53.315054 32629 consensus_queue.cc:1035] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:53.320206 1804 tablet_bootstrap.cc:492] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap starting.
I20240503 14:13:53.322306 31072 raft_consensus.cc:1261] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:53.322996 1850 consensus_queue.cc:1035] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:53.325598 31071 raft_consensus.cc:1261] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:53.326318 32629 consensus_queue.cc:1035] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:53.331187 1882 tablet_copy_client.cc:308] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44297
I20240503 14:13:53.332059 31221 tablet_copy_service.cc:140] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Received BeginTabletCopySession request for tablet f17feb9b64214eaebcbe31dc45c75756 from peer 6967bdda50404d68a5ac043b709440e1 ({username='slave'} at 127.22.242.69:38549)
I20240503 14:13:53.332255 31221 tablet_copy_service.cc:161] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Beginning new tablet copy session on tablet f17feb9b64214eaebcbe31dc45c75756 from peer 6967bdda50404d68a5ac043b709440e1 at {username='slave'} at 127.22.242.69:38549: session id = 6967bdda50404d68a5ac043b709440e1-f17feb9b64214eaebcbe31dc45c75756
I20240503 14:13:53.335875 1787 log.cc:824] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:53.338320 31221 tablet_copy_source_session.cc:205] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:53.339933 1882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f17feb9b64214eaebcbe31dc45c75756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.342433 1850 raft_consensus.cc:2932] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:53.352552 1817 tablet_copy_client.cc:647] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:53.355916 31201 raft_consensus.cc:2932] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:53.359614 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:53.368893 31072 raft_consensus.cc:2932] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:53.368111 30794 catalog_manager.cc:5484] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:53.375594 1817 tablet_copy_client.cc:523] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:53.380251 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:53.388069 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:53.396978 31071 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:53.402515 1882 tablet_copy_client.cc:783] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:53.402756 1882 tablet_copy_client.cc:647] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:53.414824 1882 tablet_copy_client.cc:523] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:53.418952 1882 tablet_bootstrap.cc:492] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1: Bootstrap starting.
I20240503 14:13:53.424629 1920 ts_tablet_manager.cc:902] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5: Initiating tablet copy from peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:53.394019 1914 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:53.466434 1827 tablet_copy_client.cc:783] T 0f45824b1fff4302abd03c1fb48c6c7b P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:53.466985 1822 tablet_bootstrap.cc:492] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap starting.
I20240503 14:13:53.467502 1827 tablet_copy_client.cc:647] T 0f45824b1fff4302abd03c1fb48c6c7b P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:53.472609 1805 tablet_bootstrap.cc:492] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap starting.
I20240503 14:13:53.475365 1827 tablet_copy_client.cc:523] T 0f45824b1fff4302abd03c1fb48c6c7b P 4665a3377574417ba0d3165c236533c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:53.479060 30796 catalog_manager.cc:5484] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "0dc902d91ccc435a82a5ede4a7729811" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:53.481711 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:53.482214 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:53.498083 31200 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:53.515715 1851 tablet_copy_client.cc:783] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:53.516201 1763 tablet_bootstrap.cc:492] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5: Bootstrap starting.
I20240503 14:13:53.519170 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:53.524636 1851 tablet_copy_client.cc:647] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:53.527271 1851 tablet_copy_client.cc:523] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:53.532220 1763 log.cc:824] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:53.545850 1799 tablet_bootstrap.cc:492] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap starting.
I20240503 14:13:53.552250 1817 tablet_bootstrap.cc:492] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap starting.
I20240503 14:13:53.567130 1919 ts_tablet_manager.cc:902] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91: Initiating tablet copy from peer e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:53.573951 1799 log.cc:824] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:53.585857 1920 tablet_copy_client.cc:308] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.68:37063
I20240503 14:13:53.595095 31352 tablet_copy_service.cc:140] P 82cfdc130877418dbe9d980d368973be: Received BeginTabletCopySession request for tablet 626fcb185af0446b89ae82066e08981d from peer 85f69eefe4294197923dff5efadcbbd5 ({username='slave'} at 127.22.242.72:36333)
I20240503 14:13:53.595290 31352 tablet_copy_service.cc:161] P 82cfdc130877418dbe9d980d368973be: Beginning new tablet copy session on tablet 626fcb185af0446b89ae82066e08981d from peer 85f69eefe4294197923dff5efadcbbd5 at {username='slave'} at 127.22.242.72:36333: session id = 85f69eefe4294197923dff5efadcbbd5-626fcb185af0446b89ae82066e08981d
I20240503 14:13:53.598884 31352 tablet_copy_source_session.cc:205] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:53.600587 1920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 626fcb185af0446b89ae82066e08981d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.605337 1851 tablet_bootstrap.cc:492] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0: Bootstrap starting.
I20240503 14:13:53.616369 1827 tablet_bootstrap.cc:492] T 0f45824b1fff4302abd03c1fb48c6c7b P 4665a3377574417ba0d3165c236533c0: Bootstrap starting.
W20240503 14:13:53.628193 30874 consensus_peers.cc:485] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 -> Peer fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829): Couldn't send request to peer fa71cbdf0e544b81bd4e7e27196d2c91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a9b84d307b8432bbdf6b70a11bf66b9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:53.659840 1920 tablet_copy_client.cc:783] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:53.660122 1920 tablet_copy_client.cc:647] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:53.662540 1920 tablet_copy_client.cc:523] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:53.663287 508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:53.663396 507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:53.674026 31331 consensus_queue.cc:237] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:53.680384 1920 tablet_bootstrap.cc:492] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5: Bootstrap starting.
I20240503 14:13:53.722707 1851 log.cc:824] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0: Log is configured to *not* fsync() on all Append() calls
W20240503 14:13:53.743773 31268 consensus_peers.cc:485] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc77a7da906b419085258f72667ff185. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:53.744817 31200 raft_consensus.cc:1261] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:53.745328 1578 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:53.758101 30939 raft_consensus.cc:1261] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:53.758716 1578 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:53.805959 1787 tablet_bootstrap.cc:492] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:53.807281 1787 tablet_bootstrap.cc:492] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1: Bootstrap complete.
I20240503 14:13:53.812388 1787 ts_tablet_manager.cc:1367] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1: Time spent bootstrapping tablet: real 0.650s user 0.067s sys 0.001s
I20240503 14:13:53.814775 1919 tablet_copy_client.cc:308] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44297
I20240503 14:13:53.815551 31221 tablet_copy_service.cc:140] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Received BeginTabletCopySession request for tablet 91a7b8a53140429b956178d43478a541 from peer fa71cbdf0e544b81bd4e7e27196d2c91 ({username='slave'} at 127.22.242.70:33209)
I20240503 14:13:53.815723 31221 tablet_copy_service.cc:161] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Beginning new tablet copy session on tablet 91a7b8a53140429b956178d43478a541 from peer fa71cbdf0e544b81bd4e7e27196d2c91 at {username='slave'} at 127.22.242.70:33209: session id = fa71cbdf0e544b81bd4e7e27196d2c91-91a7b8a53140429b956178d43478a541
I20240503 14:13:53.818758 31221 tablet_copy_source_session.cc:205] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:53.811846 1818 tablet_bootstrap.cc:492] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:53.825925 1919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a7b8a53140429b956178d43478a541. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.826423 1818 tablet_bootstrap.cc:492] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1: Bootstrap complete.
I20240503 14:13:53.831198 1956 ts_tablet_manager.cc:902] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91: Initiating tablet copy from peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:53.831418 1818 ts_tablet_manager.cc:1367] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1: Time spent bootstrapping tablet: real 0.539s user 0.044s sys 0.009s
I20240503 14:13:53.832878 1578 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:53.834861 1787 raft_consensus.cc:348] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:53.835891 1787 raft_consensus.cc:729] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6967bdda50404d68a5ac043b709440e1, State: Initialized, Role: LEARNER
I20240503 14:13:53.835616 1818 raft_consensus.cc:348] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:53.835719 31202 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:53.836364 1818 raft_consensus.cc:729] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6967bdda50404d68a5ac043b709440e1, State: Initialized, Role: LEARNER
I20240503 14:13:53.836932 1818 consensus_queue.cc:260] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:53.837538 1787 consensus_queue.cc:260] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:53.844132 30796 catalog_manager.cc:5484] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:53.852237 1787 ts_tablet_manager.cc:1398] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1: Time spent starting tablet: real 0.039s user 0.011s sys 0.001s
I20240503 14:13:53.853372 31221 tablet_copy_service.cc:342] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Request end of tablet copy session 6967bdda50404d68a5ac043b709440e1-1bb75d7d9499475ca60c633c0801d639 received from {username='slave'} at 127.22.242.69:38549
I20240503 14:13:53.853567 31221 tablet_copy_service.cc:433] P e7badbe63a8a4d9ca0c78818c6f4c8ce: ending tablet copy session 6967bdda50404d68a5ac043b709440e1-1bb75d7d9499475ca60c633c0801d639 on tablet 1bb75d7d9499475ca60c633c0801d639 with peer 6967bdda50404d68a5ac043b709440e1
I20240503 14:13:53.855094 1818 ts_tablet_manager.cc:1398] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1: Time spent starting tablet: real 0.023s user 0.004s sys 0.001s
I20240503 14:13:53.855926 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session 6967bdda50404d68a5ac043b709440e1-63b2120788a54af78f02449f46bf23c7 received from {username='slave'} at 127.22.242.69:44021
I20240503 14:13:53.856096 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session 6967bdda50404d68a5ac043b709440e1-63b2120788a54af78f02449f46bf23c7 on tablet 63b2120788a54af78f02449f46bf23c7 with peer 6967bdda50404d68a5ac043b709440e1
I20240503 14:13:53.857532 30939 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:53.875830 1959 ts_tablet_manager.cc:902] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5: Initiating tablet copy from peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:53.889211 31332 consensus_queue.cc:237] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:53.893268 31072 raft_consensus.cc:1261] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:53.893877 1850 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:53.896503 1850 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:53.898254 1817 tablet_bootstrap.cc:492] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:53.898820 1817 tablet_bootstrap.cc:492] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap complete.
I20240503 14:13:53.899312 1817 ts_tablet_manager.cc:1367] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent bootstrapping tablet: real 0.409s user 0.050s sys 0.004s
I20240503 14:13:53.899349 1737 tablet_bootstrap.cc:492] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:53.899811 1737 tablet_bootstrap.cc:492] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1: Bootstrap complete.
I20240503 14:13:53.900101 1737 ts_tablet_manager.cc:1367] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1: Time spent bootstrapping tablet: real 0.722s user 0.063s sys 0.000s
I20240503 14:13:53.900964 31072 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
W20240503 14:13:53.905947 31268 consensus_peers.cc:485] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:53.905820 1817 raft_consensus.cc:348] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:53.906368 1817 raft_consensus.cc:729] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa71cbdf0e544b81bd4e7e27196d2c91, State: Initialized, Role: LEARNER
I20240503 14:13:53.907379 1737 raft_consensus.cc:348] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:53.907675 1737 raft_consensus.cc:729] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6967bdda50404d68a5ac043b709440e1, State: Initialized, Role: LEARNER
I20240503 14:13:53.907893 1737 consensus_queue.cc:260] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:53.908102 1817 consensus_queue.cc:260] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:53.909138 30797 catalog_manager.cc:5484] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index 4 to 5, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:53.934634 1851 tablet_bootstrap.cc:492] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:53.937634 1737 ts_tablet_manager.cc:1398] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20240503 14:13:53.940151 1851 tablet_bootstrap.cc:492] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0: Bootstrap complete.
I20240503 14:13:53.940636 31221 tablet_copy_service.cc:342] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Request end of tablet copy session 6967bdda50404d68a5ac043b709440e1-2f82073122a74268a464387659dd90bf received from {username='slave'} at 127.22.242.69:38549
I20240503 14:13:53.940821 31221 tablet_copy_service.cc:433] P e7badbe63a8a4d9ca0c78818c6f4c8ce: ending tablet copy session 6967bdda50404d68a5ac043b709440e1-2f82073122a74268a464387659dd90bf on tablet 2f82073122a74268a464387659dd90bf with peer 6967bdda50404d68a5ac043b709440e1
I20240503 14:13:53.941540 1851 ts_tablet_manager.cc:1367] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0: Time spent bootstrapping tablet: real 0.337s user 0.061s sys 0.004s
I20240503 14:13:53.949040 1851 raft_consensus.cc:348] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:53.958321 30939 raft_consensus.cc:1261] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:53.959045 1578 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:53.964651 1851 raft_consensus.cc:729] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4665a3377574417ba0d3165c236533c0, State: Initialized, Role: LEARNER
I20240503 14:13:53.968050 1851 consensus_queue.cc:260] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:53.981478 30939 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:53.983577 1851 ts_tablet_manager.cc:1398] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0: Time spent starting tablet: real 0.042s user 0.011s sys 0.000s
I20240503 14:13:53.991672 31352 tablet_copy_service.cc:342] P 82cfdc130877418dbe9d980d368973be: Request end of tablet copy session 4665a3377574417ba0d3165c236533c0-55cf34de74694f47ad3f9f4d9bdffae4 received from {username='slave'} at 127.22.242.71:44595
I20240503 14:13:53.991865 31352 tablet_copy_service.cc:433] P 82cfdc130877418dbe9d980d368973be: ending tablet copy session 4665a3377574417ba0d3165c236533c0-55cf34de74694f47ad3f9f4d9bdffae4 on tablet 55cf34de74694f47ad3f9f4d9bdffae4 with peer 4665a3377574417ba0d3165c236533c0
I20240503 14:13:53.992851 396 raft_consensus.cc:1203] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:13:54.006258 1882 tablet_bootstrap.cc:492] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.006875 1882 tablet_bootstrap.cc:492] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1: Bootstrap complete.
I20240503 14:13:54.007227 1882 ts_tablet_manager.cc:1367] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1: Time spent bootstrapping tablet: real 0.588s user 0.053s sys 0.004s
W20240503 14:13:54.007728 31265 consensus_peers.cc:485] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be -> Peer fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829): Couldn't send request to peer fa71cbdf0e544b81bd4e7e27196d2c91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 626fcb185af0446b89ae82066e08981d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:54.008220 1882 raft_consensus.cc:348] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:54.008562 1882 raft_consensus.cc:729] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6967bdda50404d68a5ac043b709440e1, State: Initialized, Role: LEARNER
I20240503 14:13:54.008831 1882 consensus_queue.cc:260] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:54.009533 1882 ts_tablet_manager.cc:1398] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:54.010226 31221 tablet_copy_service.cc:342] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Request end of tablet copy session 6967bdda50404d68a5ac043b709440e1-f17feb9b64214eaebcbe31dc45c75756 received from {username='slave'} at 127.22.242.69:38549
I20240503 14:13:54.010316 1817 ts_tablet_manager.cc:1398] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent starting tablet: real 0.111s user 0.026s sys 0.005s
I20240503 14:13:54.010376 31221 tablet_copy_service.cc:433] P e7badbe63a8a4d9ca0c78818c6f4c8ce: ending tablet copy session 6967bdda50404d68a5ac043b709440e1-f17feb9b64214eaebcbe31dc45c75756 on tablet f17feb9b64214eaebcbe31dc45c75756 with peer 6967bdda50404d68a5ac043b709440e1
I20240503 14:13:54.012259 31221 tablet_copy_service.cc:342] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Request end of tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-43221a82a1124bf8840c6369f27f47f2 received from {username='slave'} at 127.22.242.70:33209
I20240503 14:13:54.012388 31221 tablet_copy_service.cc:433] P e7badbe63a8a4d9ca0c78818c6f4c8ce: ending tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-43221a82a1124bf8840c6369f27f47f2 on tablet 43221a82a1124bf8840c6369f27f47f2 with peer fa71cbdf0e544b81bd4e7e27196d2c91
I20240503 14:13:54.024520 1763 tablet_bootstrap.cc:492] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.025184 1763 tablet_bootstrap.cc:492] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5: Bootstrap complete.
I20240503 14:13:54.025223 1956 tablet_copy_client.cc:308] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:46013
I20240503 14:13:54.029161 1822 tablet_bootstrap.cc:492] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.032032 1919 tablet_copy_client.cc:783] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:54.032469 1763 ts_tablet_manager.cc:1367] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5: Time spent bootstrapping tablet: real 0.573s user 0.050s sys 0.014s
I20240503 14:13:54.034704 1822 tablet_bootstrap.cc:492] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap complete.
I20240503 14:13:54.035763 1822 ts_tablet_manager.cc:1367] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: Time spent bootstrapping tablet: real 0.577s user 0.060s sys 0.000s
I20240503 14:13:54.036553 1959 tablet_copy_client.cc:308] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.68:37063
I20240503 14:13:54.037325 31352 tablet_copy_service.cc:140] P 82cfdc130877418dbe9d980d368973be: Received BeginTabletCopySession request for tablet 69b741aad35d4bfc8fea5b8e21d4a733 from peer 85f69eefe4294197923dff5efadcbbd5 ({username='slave'} at 127.22.242.72:36333)
I20240503 14:13:54.037523 31352 tablet_copy_service.cc:161] P 82cfdc130877418dbe9d980d368973be: Beginning new tablet copy session on tablet 69b741aad35d4bfc8fea5b8e21d4a733 from peer 85f69eefe4294197923dff5efadcbbd5 at {username='slave'} at 127.22.242.72:36333: session id = 85f69eefe4294197923dff5efadcbbd5-69b741aad35d4bfc8fea5b8e21d4a733
I20240503 14:13:54.040879 31352 tablet_copy_source_session.cc:205] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:54.041100 1822 raft_consensus.cc:348] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:54.041651 1822 raft_consensus.cc:729] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85f69eefe4294197923dff5efadcbbd5, State: Initialized, Role: LEARNER
I20240503 14:13:54.042336 1959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b741aad35d4bfc8fea5b8e21d4a733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.042281 1822 consensus_queue.cc:260] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:54.032824 1805 tablet_bootstrap.cc:492] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.043290 1805 tablet_bootstrap.cc:492] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap complete.
I20240503 14:13:54.043597 1805 ts_tablet_manager.cc:1367] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5: Time spent bootstrapping tablet: real 0.585s user 0.058s sys 0.005s
I20240503 14:13:54.044435 1919 tablet_copy_client.cc:647] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:54.044464 1805 raft_consensus.cc:348] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:54.044739 1805 raft_consensus.cc:729] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85f69eefe4294197923dff5efadcbbd5, State: Initialized, Role: LEARNER
I20240503 14:13:54.045814 1805 consensus_queue.cc:260] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:54.046895 1919 tablet_copy_client.cc:523] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:54.046919 660 raft_consensus.cc:1203] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:54.051927 1920 tablet_bootstrap.cc:492] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.052574 1920 tablet_bootstrap.cc:492] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5: Bootstrap complete.
I20240503 14:13:54.052927 1920 ts_tablet_manager.cc:1367] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5: Time spent bootstrapping tablet: real 0.373s user 0.049s sys 0.006s
I20240503 14:13:54.053866 1920 raft_consensus.cc:348] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:54.054258 1920 raft_consensus.cc:729] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85f69eefe4294197923dff5efadcbbd5, State: Initialized, Role: LEARNER
I20240503 14:13:54.054550 1920 consensus_queue.cc:260] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:54.054558 1763 raft_consensus.cc:348] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:54.055454 1763 raft_consensus.cc:729] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85f69eefe4294197923dff5efadcbbd5, State: Initialized, Role: LEARNER
I20240503 14:13:54.055784 1763 consensus_queue.cc:260] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:54.071090 31072 consensus_queue.cc:237] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:54.077734 1822 ts_tablet_manager.cc:1398] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: Time spent starting tablet: real 0.042s user 0.010s sys 0.000s
I20240503 14:13:54.082165 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session 85f69eefe4294197923dff5efadcbbd5-63b2120788a54af78f02449f46bf23c7 received from {username='slave'} at 127.22.242.72:54703
I20240503 14:13:54.082362 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session 85f69eefe4294197923dff5efadcbbd5-63b2120788a54af78f02449f46bf23c7 on tablet 63b2120788a54af78f02449f46bf23c7 with peer 85f69eefe4294197923dff5efadcbbd5
I20240503 14:13:54.085757 1920 ts_tablet_manager.cc:1398] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5: Time spent starting tablet: real 0.033s user 0.002s sys 0.014s
I20240503 14:13:54.086553 31352 tablet_copy_service.cc:342] P 82cfdc130877418dbe9d980d368973be: Request end of tablet copy session 85f69eefe4294197923dff5efadcbbd5-626fcb185af0446b89ae82066e08981d received from {username='slave'} at 127.22.242.72:36333
I20240503 14:13:54.086776 31352 tablet_copy_service.cc:433] P 82cfdc130877418dbe9d980d368973be: ending tablet copy session 85f69eefe4294197923dff5efadcbbd5-626fcb185af0446b89ae82066e08981d on tablet 626fcb185af0446b89ae82066e08981d with peer 85f69eefe4294197923dff5efadcbbd5
I20240503 14:13:54.087003 1805 ts_tablet_manager.cc:1398] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5: Time spent starting tablet: real 0.043s user 0.000s sys 0.010s
I20240503 14:13:54.087783 31221 tablet_copy_service.cc:342] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Request end of tablet copy session 85f69eefe4294197923dff5efadcbbd5-1bb75d7d9499475ca60c633c0801d639 received from {username='slave'} at 127.22.242.72:40123
I20240503 14:13:54.087911 31221 tablet_copy_service.cc:433] P e7badbe63a8a4d9ca0c78818c6f4c8ce: ending tablet copy session 85f69eefe4294197923dff5efadcbbd5-1bb75d7d9499475ca60c633c0801d639 on tablet 1bb75d7d9499475ca60c633c0801d639 with peer 85f69eefe4294197923dff5efadcbbd5
I20240503 14:13:54.089114 1763 ts_tablet_manager.cc:1398] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5: Time spent starting tablet: real 0.055s user 0.016s sys 0.004s
I20240503 14:13:54.090054 31221 tablet_copy_service.cc:342] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Request end of tablet copy session 85f69eefe4294197923dff5efadcbbd5-2f82073122a74268a464387659dd90bf received from {username='slave'} at 127.22.242.72:40123
I20240503 14:13:54.090186 31221 tablet_copy_service.cc:433] P e7badbe63a8a4d9ca0c78818c6f4c8ce: ending tablet copy session 85f69eefe4294197923dff5efadcbbd5-2f82073122a74268a464387659dd90bf on tablet 2f82073122a74268a464387659dd90bf with peer 85f69eefe4294197923dff5efadcbbd5
I20240503 14:13:54.092676 1919 tablet_bootstrap.cc:492] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap starting.
I20240503 14:13:54.095693 30961 tablet_copy_service.cc:140] P 0dc902d91ccc435a82a5ede4a7729811: Received BeginTabletCopySession request for tablet 1a9b84d307b8432bbdf6b70a11bf66b9 from peer fa71cbdf0e544b81bd4e7e27196d2c91 ({username='slave'} at 127.22.242.70:54795)
I20240503 14:13:54.095947 30961 tablet_copy_service.cc:161] P 0dc902d91ccc435a82a5ede4a7729811: Beginning new tablet copy session on tablet 1a9b84d307b8432bbdf6b70a11bf66b9 from peer fa71cbdf0e544b81bd4e7e27196d2c91 at {username='slave'} at 127.22.242.70:54795: session id = fa71cbdf0e544b81bd4e7e27196d2c91-1a9b84d307b8432bbdf6b70a11bf66b9
I20240503 14:13:54.099790 30961 tablet_copy_source_session.cc:205] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 14:13:54.101430 31008 consensus_peers.cc:485] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189): Couldn't send request to peer 6967bdda50404d68a5ac043b709440e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c86ee72320b647eb84e376f8995b55d2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:54.103431 1956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a9b84d307b8432bbdf6b70a11bf66b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.103569 31200 raft_consensus.cc:1261] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:54.104225 1600 consensus_queue.cc:1035] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:54.108816 31332 raft_consensus.cc:1261] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:54.109436 1600 consensus_queue.cc:1035] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:54.111862 1956 tablet_copy_client.cc:783] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:54.112257 1956 tablet_copy_client.cc:647] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:54.119829 1956 tablet_copy_client.cc:523] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:54.151547 1959 tablet_copy_client.cc:783] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:54.152050 1959 tablet_copy_client.cc:647] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:54.175004 1959 tablet_copy_client.cc:523] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:54.181496 1600 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:54.185364 31331 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:54.187319 1959 tablet_bootstrap.cc:492] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap starting.
I20240503 14:13:54.194944 30794 catalog_manager.cc:5484] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:54.202991 1956 tablet_bootstrap.cc:492] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap starting.
I20240503 14:13:54.219738 31202 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:54.224857 396 raft_consensus.cc:1203] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:54.237437 528 raft_consensus.cc:1203] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:54.244755 1804 tablet_bootstrap.cc:492] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.245306 1804 tablet_bootstrap.cc:492] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap complete.
I20240503 14:13:54.245671 1804 ts_tablet_manager.cc:1367] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent bootstrapping tablet: real 0.926s user 0.060s sys 0.011s
I20240503 14:13:54.246628 1804 raft_consensus.cc:348] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:54.246958 1804 raft_consensus.cc:729] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa71cbdf0e544b81bd4e7e27196d2c91, State: Initialized, Role: LEARNER
I20240503 14:13:54.247215 1804 consensus_queue.cc:260] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:54.247993 1804 ts_tablet_manager.cc:1398] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:54.254962 1851 ts_tablet_manager.cc:902] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0: Initiating tablet copy from peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:54.258648 396 raft_consensus.cc:1203] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:13:54.276235 791 raft_consensus.cc:1203] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:13:54.277401 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-e7b5dd5eb46a45989bf9ffe3144bc189 received from {username='slave'} at 127.22.242.70:40181
I20240503 14:13:54.277608 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-e7b5dd5eb46a45989bf9ffe3144bc189 on tablet e7b5dd5eb46a45989bf9ffe3144bc189 with peer fa71cbdf0e544b81bd4e7e27196d2c91
I20240503 14:13:54.289597 396 raft_consensus.cc:1203] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:13:54.309149 528 raft_consensus.cc:1203] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:54.318238 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:54.339577 31332 consensus_queue.cc:237] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
W20240503 14:13:54.342288 31268 consensus_peers.cc:485] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6234489254fc4882a39b7df28ddf673f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:54.342329 31072 raft_consensus.cc:1261] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:54.342907 1578 consensus_queue.cc:1035] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:54.345772 30939 raft_consensus.cc:1261] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:54.348536 1578 consensus_queue.cc:1035] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:54.352316 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:54.371625 1851 tablet_copy_client.cc:308] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.68:37063
I20240503 14:13:54.374214 1850 raft_consensus.cc:2932] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:54.374873 31071 raft_consensus.cc:2932] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:54.376118 31352 tablet_copy_service.cc:140] P 82cfdc130877418dbe9d980d368973be: Received BeginTabletCopySession request for tablet bc77a7da906b419085258f72667ff185 from peer 4665a3377574417ba0d3165c236533c0 ({username='slave'} at 127.22.242.71:44595)
I20240503 14:13:54.376361 31352 tablet_copy_service.cc:161] P 82cfdc130877418dbe9d980d368973be: Beginning new tablet copy session on tablet bc77a7da906b419085258f72667ff185 from peer 4665a3377574417ba0d3165c236533c0 at {username='slave'} at 127.22.242.71:44595: session id = 4665a3377574417ba0d3165c236533c0-bc77a7da906b419085258f72667ff185
I20240503 14:13:54.382901 30792 catalog_manager.cc:5484] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:54.389720 31352 tablet_copy_source_session.cc:205] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:54.402624 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:54.403982 1851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc77a7da906b419085258f72667ff185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.411375 791 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:54.432287 507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:54.448514 30939 raft_consensus.cc:2932] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:54.460351 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:54.461462 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:54.465670 1804 ts_tablet_manager.cc:902] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91: Initiating tablet copy from peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:54.468128 1804 tablet_copy_client.cc:308] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.68:37063
I20240503 14:13:54.479131 31352 tablet_copy_service.cc:140] P 82cfdc130877418dbe9d980d368973be: Received BeginTabletCopySession request for tablet 626fcb185af0446b89ae82066e08981d from peer fa71cbdf0e544b81bd4e7e27196d2c91 ({username='slave'} at 127.22.242.70:37791)
I20240503 14:13:54.479348 31352 tablet_copy_service.cc:161] P 82cfdc130877418dbe9d980d368973be: Beginning new tablet copy session on tablet 626fcb185af0446b89ae82066e08981d from peer fa71cbdf0e544b81bd4e7e27196d2c91 at {username='slave'} at 127.22.242.70:37791: session id = fa71cbdf0e544b81bd4e7e27196d2c91-626fcb185af0446b89ae82066e08981d
I20240503 14:13:54.482378 31352 tablet_copy_source_session.cc:205] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:54.487365 1804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 626fcb185af0446b89ae82066e08981d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.489112 1799 tablet_bootstrap.cc:492] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.489689 1799 tablet_bootstrap.cc:492] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap complete.
I20240503 14:13:54.490084 1799 ts_tablet_manager.cc:1367] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent bootstrapping tablet: real 0.956s user 0.056s sys 0.003s
I20240503 14:13:54.491089 1799 raft_consensus.cc:348] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:54.491397 1799 raft_consensus.cc:729] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa71cbdf0e544b81bd4e7e27196d2c91, State: Initialized, Role: LEARNER
I20240503 14:13:54.491663 1799 consensus_queue.cc:260] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:54.494906 1804 tablet_copy_client.cc:783] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:54.495180 1804 tablet_copy_client.cc:647] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:54.497062 1804 tablet_copy_client.cc:523] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:54.500613 1804 tablet_bootstrap.cc:492] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap starting.
I20240503 14:13:54.512221 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:54.518848 1827 tablet_bootstrap.cc:492] T 0f45824b1fff4302abd03c1fb48c6c7b P 4665a3377574417ba0d3165c236533c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.523939 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:54.534116 1850 raft_consensus.cc:1050] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be: attempting to promote NON_VOTER 4665a3377574417ba0d3165c236533c0 to VOTER
I20240503 14:13:54.535156 1850 consensus_queue.cc:237] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:13:54.544385 1827 tablet_bootstrap.cc:492] T 0f45824b1fff4302abd03c1fb48c6c7b P 4665a3377574417ba0d3165c236533c0: Bootstrap complete.
I20240503 14:13:54.544828 1827 ts_tablet_manager.cc:1367] T 0f45824b1fff4302abd03c1fb48c6c7b P 4665a3377574417ba0d3165c236533c0: Time spent bootstrapping tablet: real 0.929s user 0.062s sys 0.002s
I20240503 14:13:54.546044 1827 raft_consensus.cc:348] T 0f45824b1fff4302abd03c1fb48c6c7b P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:54.546384 1827 raft_consensus.cc:729] T 0f45824b1fff4302abd03c1fb48c6c7b P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4665a3377574417ba0d3165c236533c0, State: Initialized, Role: LEARNER
I20240503 14:13:54.546602 1827 consensus_queue.cc:260] T 0f45824b1fff4302abd03c1fb48c6c7b P 4665a3377574417ba0d3165c236533c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:54.550375 791 raft_consensus.cc:1203] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:13:54.551357 1827 ts_tablet_manager.cc:1398] T 0f45824b1fff4302abd03c1fb48c6c7b P 4665a3377574417ba0d3165c236533c0: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:13:54.555429 1851 tablet_copy_client.cc:783] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:54.555732 1851 tablet_copy_client.cc:647] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:54.555903 31352 tablet_copy_service.cc:342] P 82cfdc130877418dbe9d980d368973be: Request end of tablet copy session 4665a3377574417ba0d3165c236533c0-0f45824b1fff4302abd03c1fb48c6c7b received from {username='slave'} at 127.22.242.71:44595
I20240503 14:13:54.556078 31352 tablet_copy_service.cc:433] P 82cfdc130877418dbe9d980d368973be: ending tablet copy session 4665a3377574417ba0d3165c236533c0-0f45824b1fff4302abd03c1fb48c6c7b on tablet 0f45824b1fff4302abd03c1fb48c6c7b with peer 4665a3377574417ba0d3165c236533c0
I20240503 14:13:54.557644 1851 tablet_copy_client.cc:523] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:54.561604 1851 tablet_bootstrap.cc:492] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0: Bootstrap starting.
I20240503 14:13:54.581974 791 raft_consensus.cc:1203] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:13:54.583001 31072 consensus_queue.cc:237] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
W20240503 14:13:54.588076 31008 consensus_peers.cc:485] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3e80559017d4a6ba16d34e0b7aab085. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:54.589280 31200 raft_consensus.cc:1261] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:54.590871 1602 consensus_queue.cc:1035] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:54.591006 31332 raft_consensus.cc:1261] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:54.591697 1602 consensus_queue.cc:1035] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:54.594139 1537 raft_consensus.cc:1050] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER 6967bdda50404d68a5ac043b709440e1 to VOTER
I20240503 14:13:54.595629 1537 consensus_queue.cc:237] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:13:54.599748 1799 ts_tablet_manager.cc:1398] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent starting tablet: real 0.110s user 0.002s sys 0.000s
I20240503 14:13:54.604539 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-c837def6d76242f08b20a448541e64d6 received from {username='slave'} at 127.22.242.70:40181
I20240503 14:13:54.604714 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-c837def6d76242f08b20a448541e64d6 on tablet c837def6d76242f08b20a448541e64d6 with peer fa71cbdf0e544b81bd4e7e27196d2c91
I20240503 14:13:54.610863 660 raft_consensus.cc:1261] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:54.611621 1722 consensus_queue.cc:1035] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:54.612915 1959 tablet_bootstrap.cc:492] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.613344 1959 tablet_bootstrap.cc:492] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap complete.
I20240503 14:13:54.613639 1959 ts_tablet_manager.cc:1367] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5: Time spent bootstrapping tablet: real 0.426s user 0.044s sys 0.013s
I20240503 14:13:54.614435 1959 raft_consensus.cc:348] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:54.614701 1959 raft_consensus.cc:729] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85f69eefe4294197923dff5efadcbbd5, State: Initialized, Role: LEARNER
I20240503 14:13:54.614914 1959 consensus_queue.cc:260] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:54.623294 31200 raft_consensus.cc:1261] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:54.623651 1600 raft_consensus.cc:2932] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:54.623952 1722 consensus_queue.cc:1035] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:54.628887 31201 raft_consensus.cc:2932] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:54.635803 30797 catalog_manager.cc:5484] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:54.642088 30939 raft_consensus.cc:1261] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:54.642750 1722 consensus_queue.cc:1035] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:54.644243 1959 ts_tablet_manager.cc:1398] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5: Time spent starting tablet: real 0.030s user 0.002s sys 0.001s
I20240503 14:13:54.644905 31352 tablet_copy_service.cc:342] P 82cfdc130877418dbe9d980d368973be: Request end of tablet copy session 85f69eefe4294197923dff5efadcbbd5-69b741aad35d4bfc8fea5b8e21d4a733 received from {username='slave'} at 127.22.242.72:36333
I20240503 14:13:54.645023 31352 tablet_copy_service.cc:433] P 82cfdc130877418dbe9d980d368973be: ending tablet copy session 85f69eefe4294197923dff5efadcbbd5-69b741aad35d4bfc8fea5b8e21d4a733 on tablet 69b741aad35d4bfc8fea5b8e21d4a733 with peer 85f69eefe4294197923dff5efadcbbd5
I20240503 14:13:54.668859 2070 ts_tablet_manager.cc:902] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:54.674165 31332 raft_consensus.cc:2932] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:54.678575 396 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:54.679268 1600 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:54.679435 791 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:54.679848 1578 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:54.681385 30941 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:54.684185 31331 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:54.685084 1575 raft_consensus.cc:1050] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce: attempting to promote NON_VOTER 6967bdda50404d68a5ac043b709440e1 to VOTER
I20240503 14:13:54.686201 1575 consensus_queue.cc:237] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:13:54.688767 2079 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:54.689955 31331 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:54.690495 1628 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:54.691228 2079 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:54.692185 30938 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:54.692274 2079 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:54.692207 30940 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:54.697393 395 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:54.698661 791 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:54.699651 1631 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:54.700878 1628 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:54.701340 1628 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:54.705598 1851 tablet_bootstrap.cc:492] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.706105 1851 tablet_bootstrap.cc:492] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0: Bootstrap complete.
I20240503 14:13:54.706405 1851 ts_tablet_manager.cc:1367] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0: Time spent bootstrapping tablet: real 0.145s user 0.052s sys 0.001s
I20240503 14:13:54.689797 31202 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:54.721659 30794 catalog_manager.cc:5484] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:54.723393 1851 raft_consensus.cc:348] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:54.723723 1851 raft_consensus.cc:729] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4665a3377574417ba0d3165c236533c0, State: Initialized, Role: LEARNER
I20240503 14:13:54.723963 1851 consensus_queue.cc:260] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:54.724633 1851 ts_tablet_manager.cc:1398] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20240503 14:13:54.726194 31352 tablet_copy_service.cc:342] P 82cfdc130877418dbe9d980d368973be: Request end of tablet copy session 4665a3377574417ba0d3165c236533c0-bc77a7da906b419085258f72667ff185 received from {username='slave'} at 127.22.242.71:44595
I20240503 14:13:54.726356 31352 tablet_copy_service.cc:433] P 82cfdc130877418dbe9d980d368973be: ending tablet copy session 4665a3377574417ba0d3165c236533c0-bc77a7da906b419085258f72667ff185 on tablet bc77a7da906b419085258f72667ff185 with peer 4665a3377574417ba0d3165c236533c0
I20240503 14:13:54.727056 1602 raft_consensus.cc:1013] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: attempt to promote peer 85f69eefe4294197923dff5efadcbbd5: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 14:13:54.725299 660 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:54.760367 31330 consensus_queue.cc:237] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:13:54.766007 30939 raft_consensus.cc:1261] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:54.767459 2077 consensus_queue.cc:1035] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:54.780465 1537 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:54.784843 2077 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
W20240503 14:13:54.788336 1956 log.cc:924] Time spent T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
I20240503 14:13:54.789290 30939 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:54.791618 791 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:54.800354 1631 raft_consensus.cc:1013] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: attempt to promote peer 85f69eefe4294197923dff5efadcbbd5: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 14:13:54.800613 1631 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:54.805872 30792 catalog_manager.cc:5484] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 5 to 6, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:54.805085 30938 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:54.811084 659 raft_consensus.cc:1261] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:54.812032 1578 consensus_queue.cc:1035] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20240503 14:13:54.813409 396 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:54.815160 659 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:54.818382 31332 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:54.820780 1851 ts_tablet_manager.cc:902] T 6234489254fc4882a39b7df28ddf673f P 4665a3377574417ba0d3165c236533c0: Initiating tablet copy from peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
W20240503 14:13:54.831000 31135 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.65:46013, user_credentials={real_user=slave}} blocked reactor thread for 46125.8us
I20240503 14:13:54.834594 2090 raft_consensus.cc:1050] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce: attempting to promote NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 to VOTER
I20240503 14:13:54.834455 31331 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:54.836033 2090 consensus_queue.cc:237] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:13:54.838740 30941 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:54.840437 30796 catalog_manager.cc:5484] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index 5 to 6, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:54.840611 31072 consensus_queue.cc:237] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:13:54.846092 2079 raft_consensus.cc:1050] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 to VOTER
I20240503 14:13:54.846268 30937 raft_consensus.cc:1261] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:54.846962 2079 consensus_queue.cc:237] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:13:54.847903 528 raft_consensus.cc:1261] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:54.848920 2089 consensus_queue.cc:1035] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 14:13:54.850001 2089 consensus_queue.cc:1035] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:54.856590 31329 raft_consensus.cc:1261] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:54.857259 1628 consensus_queue.cc:1035] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:54.864077 31330 raft_consensus.cc:1261] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:54.864094 527 raft_consensus.cc:1261] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:54.864801 2079 consensus_queue.cc:1035] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:54.865693 2079 consensus_queue.cc:1035] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:54.870471 31200 raft_consensus.cc:1261] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:54.871331 1537 consensus_queue.cc:1035] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:54.877103 30796 catalog_manager.cc:5484] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0 reported cstate change: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:54.888319 31180 tablet_service.cc:1514] Processing DeleteTablet for tablet 55cf34de74694f47ad3f9f4d9bdffae4 with delete_type TABLET_DATA_TOMBSTONED (TS e7badbe63a8a4d9ca0c78818c6f4c8ce not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45854
I20240503 14:13:54.888629 30786 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 55cf34de74694f47ad3f9f4d9bdffae4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:54.898233 1575 raft_consensus.cc:1050] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce: attempting to promote NON_VOTER 6967bdda50404d68a5ac043b709440e1 to VOTER
I20240503 14:13:54.899343 1575 consensus_queue.cc:237] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:54.898211 1628 raft_consensus.cc:1050] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce: attempting to promote NON_VOTER 6967bdda50404d68a5ac043b709440e1 to VOTER
I20240503 14:13:54.903681 2070 tablet_copy_client.cc:308] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:13:54.904366 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet c86ee72320b647eb84e376f8995b55d2 from peer 6967bdda50404d68a5ac043b709440e1 ({username='slave'} at 127.22.242.69:44021)
I20240503 14:13:54.904359 1628 consensus_queue.cc:237] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:13:54.904577 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet c86ee72320b647eb84e376f8995b55d2 from peer 6967bdda50404d68a5ac043b709440e1 at {username='slave'} at 127.22.242.69:44021: session id = 6967bdda50404d68a5ac043b709440e1-c86ee72320b647eb84e376f8995b55d2
I20240503 14:13:54.906337 2114 tablet_replica.cc:330] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce: stopping tablet replica
I20240503 14:13:54.907929 31092 tablet_copy_source_session.cc:205] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:54.908850 2114 raft_consensus.cc:2229] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:54.909278 2114 raft_consensus.cc:2258] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:54.909164 1537 raft_consensus.cc:2932] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:54.914021 30940 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:54.915050 1537 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:54.915251 2070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86ee72320b647eb84e376f8995b55d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.915956 2114 ts_tablet_manager.cc:1875] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:54.921343 30796 catalog_manager.cc:5484] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:54.923938 1631 raft_consensus.cc:2932] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:54.925480 526 raft_consensus.cc:2932] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:54.926301 394 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:54.930073 31330 raft_consensus.cc:2932] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:54.933310 30940 raft_consensus.cc:2932] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:54.934459 31202 raft_consensus.cc:2932] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:54.934975 31072 consensus_queue.cc:237] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:13:54.934857 526 raft_consensus.cc:2932] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:54.922248 1956 tablet_bootstrap.cc:492] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.945430 1956 tablet_bootstrap.cc:492] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap complete.
I20240503 14:13:54.946075 1578 raft_consensus.cc:1050] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be: attempting to promote NON_VOTER 85f69eefe4294197923dff5efadcbbd5 to VOTER
I20240503 14:13:54.946960 1578 consensus_queue.cc:237] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:54.946738 791 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:54.947816 1956 ts_tablet_manager.cc:1367] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent bootstrapping tablet: real 0.745s user 0.056s sys 0.003s
I20240503 14:13:54.949129 528 raft_consensus.cc:1261] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:54.951329 31328 raft_consensus.cc:2932] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:54.952934 1956 raft_consensus.cc:348] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:54.957947 31072 consensus_queue.cc:237] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:54.959298 1956 raft_consensus.cc:729] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa71cbdf0e544b81bd4e7e27196d2c91, State: Initialized, Role: LEARNER
I20240503 14:13:54.959375 31071 raft_consensus.cc:1261] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:54.959677 792 raft_consensus.cc:1261] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:54.960237 526 raft_consensus.cc:1203] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:54.960307 1722 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:54.960768 1722 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:13:54.962395 31265 consensus_peers.cc:485] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be -> Peer fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829): Couldn't send request to peer fa71cbdf0e544b81bd4e7e27196d2c91. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:54.965036 1537 consensus_queue.cc:1035] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:54.966326 30794 catalog_manager.cc:5484] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:54.967759 31331 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:54.968828 1537 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:54.970633 660 raft_consensus.cc:1203] T 0f45824b1fff4302abd03c1fb48c6c7b P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:54.973538 31202 consensus_queue.cc:237] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:13:54.974362 2069 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:54.979228 31331 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:54.981366 1602 raft_consensus.cc:2932] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:54.981912 792 raft_consensus.cc:1203] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:54.982008 1850 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:54.986733 31200 consensus_queue.cc:237] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:13:54.986965 792 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:54.988584 2070 tablet_copy_client.cc:783] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:54.988835 2070 tablet_copy_client.cc:647] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:54.991092 30787 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e7b5dd5eb46a45989bf9ffe3144bc189 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:54.991286 31328 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:54.992094 2094 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:54.992962 31071 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:54.993515 30793 catalog_manager.cc:5484] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5 reported cstate change: config changed from index 5 to 6, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:54.993932 30794 catalog_manager.cc:5484] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:54.995977 30787 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 63b2120788a54af78f02449f46bf23c7 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:54.996791 527 raft_consensus.cc:2932] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:54.996968 30940 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.000363 393 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:55.002864 31328 raft_consensus.cc:1261] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:55.003496 2096 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:55.004374 526 raft_consensus.cc:1261] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:55.005152 30940 raft_consensus.cc:1261] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:55.005299 2096 consensus_queue.cc:1035] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:55.005745 2094 consensus_queue.cc:1035] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 14:13:55.007442 1851 tablet_copy_client.cc:308] T 6234489254fc4882a39b7df28ddf673f P 4665a3377574417ba0d3165c236533c0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.68:37063
I20240503 14:13:55.008188 31352 tablet_copy_service.cc:140] P 82cfdc130877418dbe9d980d368973be: Received BeginTabletCopySession request for tablet 6234489254fc4882a39b7df28ddf673f from peer 4665a3377574417ba0d3165c236533c0 ({username='slave'} at 127.22.242.71:44595)
I20240503 14:13:55.008347 31352 tablet_copy_service.cc:161] P 82cfdc130877418dbe9d980d368973be: Beginning new tablet copy session on tablet 6234489254fc4882a39b7df28ddf673f from peer 4665a3377574417ba0d3165c236533c0 at {username='slave'} at 127.22.242.71:44595: session id = 4665a3377574417ba0d3165c236533c0-6234489254fc4882a39b7df28ddf673f
I20240503 14:13:55.009719 31180 tablet_service.cc:1514] Processing DeleteTablet for tablet e7b5dd5eb46a45989bf9ffe3144bc189 with delete_type TABLET_DATA_TOMBSTONED (TS e7badbe63a8a4d9ca0c78818c6f4c8ce not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45854
I20240503 14:13:55.009967 393 raft_consensus.cc:1261] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.011392 30939 raft_consensus.cc:1261] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.011441 31352 tablet_copy_source_session.cc:205] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:55.012059 2096 consensus_queue.cc:1035] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:55.014468 792 raft_consensus.cc:1261] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:55.015101 1851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6234489254fc4882a39b7df28ddf673f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.016247 2070 tablet_copy_client.cc:523] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:55.019186 31070 raft_consensus.cc:1261] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:54.961105 1956 consensus_queue.cc:260] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:55.021703 1956 ts_tablet_manager.cc:1398] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent starting tablet: real 0.072s user 0.002s sys 0.000s
I20240503 14:13:55.022616 2096 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:55.022768 31329 raft_consensus.cc:1261] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:55.023643 30938 raft_consensus.cc:1261] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:55.023440 30793 catalog_manager.cc:5484] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) evicted. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.024068 1537 consensus_queue.cc:1035] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:55.024295 2126 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:55.024394 2094 raft_consensus.cc:2932] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.025080 31331 raft_consensus.cc:1261] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.025620 1575 consensus_queue.cc:1035] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:55.025902 30936 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:55.027426 1575 consensus_queue.cc:1035] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:55.028645 30937 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:55.031466 30936 raft_consensus.cc:2932] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.032858 395 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:55.034420 2079 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:55.038514 1631 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:55.039085 1631 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:55.039482 30961 tablet_copy_service.cc:342] P 0dc902d91ccc435a82a5ede4a7729811: Request end of tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-1a9b84d307b8432bbdf6b70a11bf66b9 received from {username='slave'} at 127.22.242.70:54795
I20240503 14:13:55.039695 30961 tablet_copy_service.cc:433] P 0dc902d91ccc435a82a5ede4a7729811: ending tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-1a9b84d307b8432bbdf6b70a11bf66b9 on tablet 1a9b84d307b8432bbdf6b70a11bf66b9 with peer fa71cbdf0e544b81bd4e7e27196d2c91
I20240503 14:13:55.040446 2090 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:55.043648 30785 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 43221a82a1124bf8840c6369f27f47f2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:55.045173 528 raft_consensus.cc:2932] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.052714 2089 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.059996 31329 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.061870 31331 raft_consensus.cc:2932] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.062932 30793 catalog_manager.cc:5484] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.063910 31332 consensus_queue.cc:237] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:13:55.067305 30785 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2f82073122a74268a464387659dd90bf with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:55.067894 790 raft_consensus.cc:1261] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:55.071378 394 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.072315 30936 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.089246 2114 ts_tablet_manager.cc:1888] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:55.090269 31202 raft_consensus.cc:1261] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:55.096482 2114 log.cc:1196] T 55cf34de74694f47ad3f9f4d9bdffae4 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/55cf34de74694f47ad3f9f4d9bdffae4
I20240503 14:13:55.097183 1537 consensus_queue.cc:1035] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:55.097636 30792 catalog_manager.cc:5484] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) evicted. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.091666 1575 raft_consensus.cc:1013] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: attempt to promote peer 85f69eefe4294197923dff5efadcbbd5: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 14:13:55.103309 393 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.108758 396 raft_consensus.cc:1261] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:55.108919 31200 consensus_queue.cc:237] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:13:55.112013 1575 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:55.114218 30785 catalog_manager.cc:4830] TS e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297): tablet 55cf34de74694f47ad3f9f4d9bdffae4 (table rebalance_test_table_5 [id=04f8dfb749c04486895d6b0f7c206446]) successfully deleted
I20240503 14:13:55.116580 2070 tablet_bootstrap.cc:492] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1: Bootstrap starting.
I20240503 14:13:55.116577 2114 tablet_replica.cc:330] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce: stopping tablet replica
I20240503 14:13:55.116852 2114 raft_consensus.cc:2229] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:55.118906 2114 raft_consensus.cc:2258] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.120010 2114 ts_tablet_manager.cc:1875] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:55.123519 31070 consensus_queue.cc:237] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:55.126621 2077 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
W20240503 14:13:55.130534 30875 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.67:44297, user_credentials={real_user=slave}} blocked reactor thread for 49436.3us
I20240503 14:13:55.131623 30936 raft_consensus.cc:1261] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:55.132131 2079 raft_consensus.cc:2932] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:55.135473 2069 consensus_queue.cc:1035] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:55.138384 30940 raft_consensus.cc:1261] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:55.138540 31202 raft_consensus.cc:2932] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:55.139020 1602 consensus_queue.cc:1035] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:13:55.139329 31005 consensus_peers.cc:485] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829): Couldn't send request to peer fa71cbdf0e544b81bd4e7e27196d2c91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 584769402a7e4e8a985a8f2363e2b8b9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:55.154834 528 raft_consensus.cc:1203] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:55.160286 396 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:55.161037 1575 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:55.163880 1631 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.164546 392 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.167896 30936 raft_consensus.cc:2932] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:55.168290 1602 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:55.169066 1628 raft_consensus.cc:2932] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.175081 30792 catalog_manager.cc:5484] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:55.175331 30940 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:55.178624 30938 raft_consensus.cc:2932] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.180212 30785 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2f82073122a74268a464387659dd90bf with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:55.180125 30800 catalog_manager.cc:5484] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 7 to 8, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:55.182889 30794 catalog_manager.cc:5484] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:55.183559 395 raft_consensus.cc:2932] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.188594 30920 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f82073122a74268a464387659dd90bf with delete_type TABLET_DATA_TOMBSTONED (TS 0dc902d91ccc435a82a5ede4a7729811 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:56642
I20240503 14:13:55.193102 30800 catalog_manager.cc:5484] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:55.204154 31311 tablet_service.cc:1514] Processing DeleteTablet for tablet 43221a82a1124bf8840c6369f27f47f2 with delete_type TABLET_DATA_TOMBSTONED (TS 82cfdc130877418dbe9d980d368973be not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48102
I20240503 14:13:55.205476 2155 tablet_replica.cc:330] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811: stopping tablet replica
I20240503 14:13:55.206297 1628 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:55.207149 31070 raft_consensus.cc:1261] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:55.207587 2155 raft_consensus.cc:2229] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:55.207955 30936 raft_consensus.cc:1261] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:55.208518 2155 raft_consensus.cc:2258] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.209260 394 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:55.211453 2155 ts_tablet_manager.cc:1875] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:55.211622 791 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:55.212106 31070 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:55.213196 31327 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:55.214021 30794 catalog_manager.cc:5484] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 5 to 6, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20240503 14:13:55.216436 31329 raft_consensus.cc:1261] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:55.216697 2077 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20240503 14:13:55.218034 1578 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:55.219699 2069 consensus_queue.cc:1035] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:55.219746 1575 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:55.222720 1575 raft_consensus.cc:1050] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce: attempting to promote NON_VOTER 85f69eefe4294197923dff5efadcbbd5 to VOTER
I20240503 14:13:55.222978 31327 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.223506 1575 consensus_queue.cc:237] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:13:55.227916 791 raft_consensus.cc:1261] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:55.228704 1575 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:55.231065 31071 raft_consensus.cc:1261] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:55.233141 2090 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:55.235976 1631 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.236565 31071 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.239068 660 raft_consensus.cc:1203] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:55.241344 790 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.251948 31326 raft_consensus.cc:1261] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:55.251850 1850 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.252791 1631 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:55.253366 1851 tablet_copy_client.cc:783] T 6234489254fc4882a39b7df28ddf673f P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:55.253610 1851 tablet_copy_client.cc:647] T 6234489254fc4882a39b7df28ddf673f P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:55.254009 31326 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:55.256832 30786 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 626fcb185af0446b89ae82066e08981d with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:55.256762 30940 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.258111 790 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.261317 31326 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.262276 31070 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.264343 2155 ts_tablet_manager.cc:1888] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 14:13:55.268266 31328 raft_consensus.cc:2932] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.268941 2155 log.cc:1196] T 2f82073122a74268a464387659dd90bf P 0dc902d91ccc435a82a5ede4a7729811: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/2f82073122a74268a464387659dd90bf
I20240503 14:13:55.269771 2114 ts_tablet_manager.cc:1888] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:55.269922 2114 log.cc:1196] T e7b5dd5eb46a45989bf9ffe3144bc189 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/e7b5dd5eb46a45989bf9ffe3144bc189
I20240503 14:13:55.275341 30784 catalog_manager.cc:4830] TS 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): tablet 2f82073122a74268a464387659dd90bf (table rebalance_test_table_0 [id=fa8e781fa3e140b6865ce5f7b429e4c4]) successfully deleted
I20240503 14:13:55.278517 30785 catalog_manager.cc:4830] TS e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297): tablet e7b5dd5eb46a45989bf9ffe3144bc189 (table rebalance_test_table_1 [id=f6d48b36155b41d3a75fe51d7d204ea7]) successfully deleted
I20240503 14:13:55.281034 30796 catalog_manager.cc:5484] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5 reported cstate change: config changed from index 6 to 7, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.281803 30796 catalog_manager.cc:5484] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5 reported cstate change: config changed from index 6 to 7, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) evicted. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.285323 772 tablet_service.cc:1514] Processing DeleteTablet for tablet 63b2120788a54af78f02449f46bf23c7 with delete_type TABLET_DATA_TOMBSTONED (TS 85f69eefe4294197923dff5efadcbbd5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37064
I20240503 14:13:55.292423 771 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f82073122a74268a464387659dd90bf with delete_type TABLET_DATA_TOMBSTONED (TS 85f69eefe4294197923dff5efadcbbd5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37064
I20240503 14:13:55.296022 2154 ts_tablet_manager.cc:902] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:55.300871 2167 tablet_replica.cc:330] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: stopping tablet replica
I20240503 14:13:55.302639 2167 raft_consensus.cc:2229] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 14:13:55.304194 2167 raft_consensus.cc:2258] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 14:13:55.309670 393 raft_consensus.cc:1261] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:55.310514 1631 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:55.312129 393 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.314018 31330 consensus_queue.cc:237] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:13:55.314498 2167 ts_tablet_manager.cc:1875] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:55.317193 1919 tablet_bootstrap.cc:492] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:55.317689 1919 tablet_bootstrap.cc:492] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap complete.
I20240503 14:13:55.317987 1919 ts_tablet_manager.cc:1367] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent bootstrapping tablet: real 1.225s user 0.055s sys 0.006s
I20240503 14:13:55.318787 792 raft_consensus.cc:1261] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:55.319406 1578 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:55.322600 1578 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.323220 1851 tablet_copy_client.cc:523] T 6234489254fc4882a39b7df28ddf673f P 4665a3377574417ba0d3165c236533c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:55.326893 30786 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 626fcb185af0446b89ae82066e08981d with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:55.327843 790 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.333057 31071 raft_consensus.cc:1261] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:55.333771 30793 catalog_manager.cc:5484] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5 reported cstate change: config changed from index 7 to 8, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.334345 1850 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:55.337656 31202 consensus_queue.cc:237] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:13:55.340394 393 raft_consensus.cc:1261] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:55.340936 30920 tablet_service.cc:1514] Processing DeleteTablet for tablet 626fcb185af0446b89ae82066e08981d with delete_type TABLET_DATA_TOMBSTONED (TS 0dc902d91ccc435a82a5ede4a7729811 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:56642
I20240503 14:13:55.340943 1631 consensus_queue.cc:1035] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:55.341415 2155 tablet_replica.cc:330] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811: stopping tablet replica
I20240503 14:13:55.341660 2155 raft_consensus.cc:2229] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:55.341737 31326 raft_consensus.cc:1261] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:55.342067 2155 raft_consensus.cc:2258] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.342206 1631 consensus_queue.cc:1035] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:55.343746 31070 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.344410 2155 ts_tablet_manager.cc:1875] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:55.351709 2155 ts_tablet_manager.cc:1888] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 14:13:55.351907 2155 log.cc:1196] T 626fcb185af0446b89ae82066e08981d P 0dc902d91ccc435a82a5ede4a7729811: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/626fcb185af0446b89ae82066e08981d
I20240503 14:13:55.351725 1631 raft_consensus.cc:2932] T f17feb9b64214eaebcbe31dc45c75756 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.355180 393 raft_consensus.cc:2932] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.356706 30785 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f17feb9b64214eaebcbe31dc45c75756 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:55.358659 1851 tablet_bootstrap.cc:492] T 6234489254fc4882a39b7df28ddf673f P 4665a3377574417ba0d3165c236533c0: Bootstrap starting.
I20240503 14:13:55.319010 1919 raft_consensus.cc:348] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:55.358878 1919 raft_consensus.cc:729] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa71cbdf0e544b81bd4e7e27196d2c91, State: Initialized, Role: LEARNER
I20240503 14:13:55.359104 1919 consensus_queue.cc:260] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:55.360939 30797 catalog_manager.cc:5484] T f17feb9b64214eaebcbe31dc45c75756 P 6967bdda50404d68a5ac043b709440e1 reported cstate change: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.363088 30784 catalog_manager.cc:4830] TS 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): tablet 626fcb185af0446b89ae82066e08981d (table rebalance_test_table_2 [id=026356e4d1494d1c8649c4b1d401565d]) successfully deleted
I20240503 14:13:55.364338 31331 raft_consensus.cc:2932] T f17feb9b64214eaebcbe31dc45c75756 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.366055 2159 tablet_replica.cc:330] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be: stopping tablet replica
I20240503 14:13:55.366526 2159 raft_consensus.cc:2229] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:55.366885 30920 tablet_service.cc:1514] Processing DeleteTablet for tablet f17feb9b64214eaebcbe31dc45c75756 with delete_type TABLET_DATA_TOMBSTONED (TS 0dc902d91ccc435a82a5ede4a7729811 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56642
I20240503 14:13:55.366966 2159 raft_consensus.cc:2258] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.367353 2155 tablet_replica.cc:330] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811: stopping tablet replica
I20240503 14:13:55.367569 2155 raft_consensus.cc:2229] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:55.367820 2155 raft_consensus.cc:2258] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.368769 2159 ts_tablet_manager.cc:1875] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:55.369073 2155 ts_tablet_manager.cc:1875] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:55.373162 2167 ts_tablet_manager.cc:1888] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:13:55.376668 1919 ts_tablet_manager.cc:1398] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent starting tablet: real 0.059s user 0.003s sys 0.001s
I20240503 14:13:55.377779 31329 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:55.378286 2167 log.cc:1196] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal/wals/63b2120788a54af78f02449f46bf23c7
I20240503 14:13:55.379323 30787 catalog_manager.cc:4830] TS 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): tablet 63b2120788a54af78f02449f46bf23c7 (table rebalance_test_table_0 [id=fa8e781fa3e140b6865ce5f7b429e4c4]) successfully deleted
I20240503 14:13:55.379894 2167 tablet_replica.cc:330] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5: stopping tablet replica
I20240503 14:13:55.380206 2167 raft_consensus.cc:2229] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 14:13:55.380474 2167 raft_consensus.cc:2258] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 14:13:55.380825 31071 consensus_queue.cc:237] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:55.381515 2167 ts_tablet_manager.cc:1875] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:55.383738 1600 raft_consensus.cc:1050] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 to VOTER
I20240503 14:13:55.384668 1600 consensus_queue.cc:237] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:13:55.390085 31221 tablet_copy_service.cc:342] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Request end of tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-91a7b8a53140429b956178d43478a541 received from {username='slave'} at 127.22.242.70:33209
I20240503 14:13:55.390242 31221 tablet_copy_service.cc:433] P e7badbe63a8a4d9ca0c78818c6f4c8ce: ending tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-91a7b8a53140429b956178d43478a541 on tablet 91a7b8a53140429b956178d43478a541 with peer fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:55.397329 31008 consensus_peers.cc:485] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebd6dc710e144e61a5fae3911f25ee5c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:55.399677 2167 ts_tablet_manager.cc:1888] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:13:55.399820 2167 log.cc:1196] T 2f82073122a74268a464387659dd90bf P 85f69eefe4294197923dff5efadcbbd5: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-7/wal/wals/2f82073122a74268a464387659dd90bf
I20240503 14:13:55.400627 30787 catalog_manager.cc:4830] TS 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): tablet 2f82073122a74268a464387659dd90bf (table rebalance_test_table_0 [id=fa8e781fa3e140b6865ce5f7b429e4c4]) successfully deleted
I20240503 14:13:55.405767 507 tablet_service.cc:1514] Processing DeleteTablet for tablet 626fcb185af0446b89ae82066e08981d with delete_type TABLET_DATA_TOMBSTONED (TS fa71cbdf0e544b81bd4e7e27196d2c91 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55796
W20240503 14:13:55.425176 31008 consensus_peers.cc:485] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5abcdc783c6144f28b9f8a9e1e162006. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:55.428305 528 raft_consensus.cc:1261] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.429116 2079 consensus_queue.cc:1035] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:55.429860 528 raft_consensus.cc:1203] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:55.431478 31202 raft_consensus.cc:1261] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:55.431718 30936 consensus_queue.cc:237] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:55.432104 2079 consensus_queue.cc:1035] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:55.436960 31329 raft_consensus.cc:1261] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.437498 2079 consensus_queue.cc:1035] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:55.442265 31200 raft_consensus.cc:1261] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:13:55.442914 30784 catalog_manager.cc:4810] TS fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829): delete failed for tablet 626fcb185af0446b89ae82066e08981d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 626fcb185af0446b89ae82066e08981d already in progress: copying tablet
I20240503 14:13:55.443315 1602 consensus_queue.cc:1035] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:55.444088 2155 ts_tablet_manager.cc:1888] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:55.446143 2154 tablet_copy_client.cc:308] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:13:55.447227 1850 raft_consensus.cc:1050] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be: attempting to promote NON_VOTER 4665a3377574417ba0d3165c236533c0 to VOTER
I20240503 14:13:55.447929 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet f3e80559017d4a6ba16d34e0b7aab085 from peer 85f69eefe4294197923dff5efadcbbd5 ({username='slave'} at 127.22.242.72:54703)
I20240503 14:13:55.448123 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet f3e80559017d4a6ba16d34e0b7aab085 from peer 85f69eefe4294197923dff5efadcbbd5 at {username='slave'} at 127.22.242.72:54703: session id = 85f69eefe4294197923dff5efadcbbd5-f3e80559017d4a6ba16d34e0b7aab085
I20240503 14:13:55.448115 1850 consensus_queue.cc:237] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:13:55.451612 31092 tablet_copy_source_session.cc:205] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:55.456619 2154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e80559017d4a6ba16d34e0b7aab085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.456593 30940 raft_consensus.cc:1261] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:55.461495 2079 consensus_queue.cc:1035] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:55.464095 2077 raft_consensus.cc:1050] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be: attempting to promote NON_VOTER 85f69eefe4294197923dff5efadcbbd5 to VOTER
I20240503 14:13:55.465276 2077 consensus_queue.cc:237] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:13:55.468086 31201 raft_consensus.cc:1261] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.469255 790 raft_consensus.cc:1261] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.470306 31070 raft_consensus.cc:1261] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.476576 2155 log.cc:1196] T f17feb9b64214eaebcbe31dc45c75756 P 0dc902d91ccc435a82a5ede4a7729811: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/f17feb9b64214eaebcbe31dc45c75756
I20240503 14:13:55.479944 2191 ts_tablet_manager.cc:902] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:55.481571 2079 raft_consensus.cc:2932] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:55.482650 31201 raft_consensus.cc:2932] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:55.489187 30797 catalog_manager.cc:5484] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:55.498989 30784 catalog_manager.cc:4830] TS 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): tablet f17feb9b64214eaebcbe31dc45c75756 (table rebalance_test_table_9 [id=acc4ce6b628f4f498f830c4572e30d65]) successfully deleted
I20240503 14:13:55.499415 2069 raft_consensus.cc:2932] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.504277 31201 raft_consensus.cc:2932] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.506203 30797 catalog_manager.cc:5484] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:55.521762 30940 raft_consensus.cc:2932] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:55.530267 527 raft_consensus.cc:2932] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.540027 31070 consensus_queue.cc:237] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:13:55.543605 31201 raft_consensus.cc:1261] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:55.544399 2079 consensus_queue.cc:1035] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:55.549576 31329 raft_consensus.cc:1261] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:55.550161 1578 consensus_queue.cc:1035] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20240503 14:13:55.552752 2079 raft_consensus.cc:2932] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.554025 2138 consensus_queue.cc:1035] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240503 14:13:55.557128 31200 raft_consensus.cc:2932] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.562556 30787 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c837def6d76242f08b20a448541e64d6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:55.564077 2126 consensus_queue.cc:1035] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20240503 14:13:55.565143 31070 raft_consensus.cc:1261] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:55.565455 1722 consensus_queue.cc:1035] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 14:13:55.566941 30793 catalog_manager.cc:5484] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.569589 660 raft_consensus.cc:1261] T 0f45824b1fff4302abd03c1fb48c6c7b P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.571257 31311 tablet_service.cc:1514] Processing DeleteTablet for tablet c837def6d76242f08b20a448541e64d6 with delete_type TABLET_DATA_TOMBSTONED (TS 82cfdc130877418dbe9d980d368973be not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48102
I20240503 14:13:55.572196 1722 consensus_queue.cc:1035] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 14:13:55.577860 2138 consensus_queue.cc:1035] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:55.580968 31202 raft_consensus.cc:1261] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.581665 30940 raft_consensus.cc:1261] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.582242 1722 consensus_queue.cc:1035] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:55.582854 1722 consensus_queue.cc:1035] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:55.586103 2159 ts_tablet_manager.cc:1888] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:55.586374 2159 log.cc:1196] T 43221a82a1124bf8840c6369f27f47f2 P 82cfdc130877418dbe9d980d368973be: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/wals/43221a82a1124bf8840c6369f27f47f2
I20240503 14:13:55.587518 2159 tablet_replica.cc:330] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be: stopping tablet replica
I20240503 14:13:55.587512 30786 catalog_manager.cc:4830] TS 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063): tablet 43221a82a1124bf8840c6369f27f47f2 (table rebalance_test_table_9 [id=acc4ce6b628f4f498f830c4572e30d65]) successfully deleted
I20240503 14:13:55.594202 30938 raft_consensus.cc:2932] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:55.603194 528 raft_consensus.cc:1261] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:55.604148 1602 consensus_queue.cc:1035] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:55.610052 30793 catalog_manager.cc:5484] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:55.615819 1914 raft_consensus.cc:2932] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:55.619575 660 raft_consensus.cc:2932] T 0f45824b1fff4302abd03c1fb48c6c7b P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:55.629321 31071 consensus_queue.cc:237] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:13:55.631866 393 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:55.633018 1600 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:55.634619 30800 catalog_manager.cc:5484] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "0dc902d91ccc435a82a5ede4a7729811" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:55.638053 527 raft_consensus.cc:2932] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.592422 1722 raft_consensus.cc:2932] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
W20240503 14:13:55.597916 2193 log.cc:924] Time spent T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be: Append to log took a long time: real 0.131s user 0.000s sys 0.000s
I20240503 14:13:55.641032 31072 raft_consensus.cc:2932] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:55.617265 31331 raft_consensus.cc:2932] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:55.641227 1602 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.645058 2159 raft_consensus.cc:2229] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:55.645015 31200 raft_consensus.cc:2932] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:55.645416 2159 pending_rounds.cc:62] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be: Trying to abort 1 pending ops.
I20240503 14:13:55.645644 2159 pending_rounds.cc:68] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023598122533089280 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "c837def6d76242f08b20a448541e64d6" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } } }
I20240503 14:13:55.645659 393 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.642695 1919 ts_tablet_manager.cc:902] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:55.653560 30787 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 63b2120788a54af78f02449f46bf23c7 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:55.660089 2191 tablet_copy_client.cc:308] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:13:55.666745 2154 tablet_copy_client.cc:783] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:55.667065 2154 tablet_copy_client.cc:647] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:55.669085 2154 tablet_copy_client.cc:523] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 14:13:55.671124 30877 consensus_peers.cc:485] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 -> Peer 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189): Couldn't send request to peer 6967bdda50404d68a5ac043b709440e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6aeaa3fe1cd4127a6144d85b0434057. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:55.681502 31201 raft_consensus.cc:2932] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.692708 31071 raft_consensus.cc:2932] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.693742 30800 catalog_manager.cc:5484] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.706483 31200 consensus_queue.cc:237] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:55.676998 31331 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:55.680119 1578 raft_consensus.cc:2932] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.715118 2154 tablet_bootstrap.cc:492] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap starting.
I20240503 14:13:55.715373 2079 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:55.717187 31331 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.691802 31326 consensus_queue.cc:237] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:13:55.724927 2159 raft_consensus.cc:2894] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 14:13:55.677271 30794 catalog_manager.cc:5484] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1 reported cstate change: config changed from index 7 to 8, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:55.726941 2159 raft_consensus.cc:2866] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 14:13:55.728130 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet ebd6dc710e144e61a5fae3911f25ee5c from peer 4665a3377574417ba0d3165c236533c0 ({username='slave'} at 127.22.242.71:41641)
I20240503 14:13:55.728351 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet ebd6dc710e144e61a5fae3911f25ee5c from peer 4665a3377574417ba0d3165c236533c0 at {username='slave'} at 127.22.242.71:41641: session id = 4665a3377574417ba0d3165c236533c0-ebd6dc710e144e61a5fae3911f25ee5c
I20240503 14:13:55.731837 31092 tablet_copy_source_session.cc:205] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:55.733457 2191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd6dc710e144e61a5fae3911f25ee5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.734337 660 raft_consensus.cc:1261] T 0f45824b1fff4302abd03c1fb48c6c7b P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:55.734308 790 raft_consensus.cc:2932] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.734957 2159 raft_consensus.cc:2258] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.743229 2191 tablet_copy_client.cc:783] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:55.743471 2191 tablet_copy_client.cc:647] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:55.742300 1578 consensus_queue.cc:1035] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20240503 14:13:55.740010 2159 ts_tablet_manager.cc:1875] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:55.747850 30940 raft_consensus.cc:1261] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:55.748510 1578 consensus_queue.cc:1035] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:55.748906 2191 tablet_copy_client.cc:523] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:55.750864 2144 raft_consensus.cc:1050] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811: attempting to promote NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 to VOTER
I20240503 14:13:55.752060 2144 consensus_queue.cc:237] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:13:55.753837 1578 raft_consensus.cc:2932] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:55.758147 30786 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f45824b1fff4302abd03c1fb48c6c7b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:55.761916 2191 tablet_bootstrap.cc:492] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0: Bootstrap starting.
I20240503 14:13:55.763159 30940 raft_consensus.cc:2932] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:55.769059 1804 tablet_bootstrap.cc:492] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:55.769531 1804 tablet_bootstrap.cc:492] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap complete.
I20240503 14:13:55.769796 1804 ts_tablet_manager.cc:1367] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent bootstrapping tablet: real 1.269s user 0.053s sys 0.009s
I20240503 14:13:55.770614 1804 raft_consensus.cc:348] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:55.770869 1804 raft_consensus.cc:729] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa71cbdf0e544b81bd4e7e27196d2c91, State: Initialized, Role: LEARNER
I20240503 14:13:55.771147 1804 consensus_queue.cc:260] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:55.773371 2077 raft_consensus.cc:1050] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be: attempting to promote NON_VOTER 4665a3377574417ba0d3165c236533c0 to VOTER
I20240503 14:13:55.774293 2077 consensus_queue.cc:237] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
W20240503 14:13:55.775151 31135 consensus_peers.cc:485] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829): Couldn't send request to peer fa71cbdf0e544b81bd4e7e27196d2c91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd5869fd8f8d41d99cf531edb7df3750. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:55.776338 31070 raft_consensus.cc:1261] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:55.776405 1804 ts_tablet_manager.cc:1398] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:13:55.776955 528 raft_consensus.cc:1261] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.777002 31352 tablet_copy_service.cc:342] P 82cfdc130877418dbe9d980d368973be: Request end of tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-626fcb185af0446b89ae82066e08981d received from {username='slave'} at 127.22.242.70:37791
I20240503 14:13:55.777158 31352 tablet_copy_service.cc:433] P 82cfdc130877418dbe9d980d368973be: ending tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-626fcb185af0446b89ae82066e08981d on tablet 626fcb185af0446b89ae82066e08981d with peer fa71cbdf0e544b81bd4e7e27196d2c91
I20240503 14:13:55.777568 1914 consensus_queue.cc:1035] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:55.778712 2090 consensus_queue.cc:1035] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:55.782176 507 tablet_service.cc:1514] Processing DeleteTablet for tablet 626fcb185af0446b89ae82066e08981d with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:55796
I20240503 14:13:55.782657 2181 tablet_replica.cc:330] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91: stopping tablet replica
I20240503 14:13:55.783233 2181 raft_consensus.cc:2229] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 14:13:55.783612 2181 raft_consensus.cc:2258] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 14:13:55.784572 30940 raft_consensus.cc:1261] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:55.784986 2181 ts_tablet_manager.cc:1875] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:55.785141 2090 consensus_queue.cc:1035] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:55.788255 31201 raft_consensus.cc:1261] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.788865 2138 consensus_queue.cc:1035] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:55.794553 2181 ts_tablet_manager.cc:1888] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:55.794782 2181 log.cc:1196] T 626fcb185af0446b89ae82066e08981d P fa71cbdf0e544b81bd4e7e27196d2c91: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal/wals/626fcb185af0446b89ae82066e08981d
I20240503 14:13:55.794592 1631 raft_consensus.cc:2932] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:55.795747 30784 catalog_manager.cc:4830] TS fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829): tablet 626fcb185af0446b89ae82066e08981d (table rebalance_test_table_2 [id=026356e4d1494d1c8649c4b1d401565d]) successfully deleted
I20240503 14:13:55.804874 30938 raft_consensus.cc:2932] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:55.806131 2138 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.807132 31200 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.811527 527 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.812904 31071 raft_consensus.cc:1261] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.813501 2138 consensus_queue.cc:1035] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:55.816011 1919 tablet_copy_client.cc:308] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:13:55.816742 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet 584769402a7e4e8a985a8f2363e2b8b9 from peer fa71cbdf0e544b81bd4e7e27196d2c91 ({username='slave'} at 127.22.242.70:40181)
I20240503 14:13:55.816908 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet 584769402a7e4e8a985a8f2363e2b8b9 from peer fa71cbdf0e544b81bd4e7e27196d2c91 at {username='slave'} at 127.22.242.70:40181: session id = fa71cbdf0e544b81bd4e7e27196d2c91-584769402a7e4e8a985a8f2363e2b8b9
I20240503 14:13:55.819195 31201 consensus_queue.cc:237] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:13:55.819864 31092 tablet_copy_source_session.cc:205] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:55.821719 1919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584769402a7e4e8a985a8f2363e2b8b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.830994 31202 raft_consensus.cc:1261] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.831734 1722 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:55.841809 30938 raft_consensus.cc:1261] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.842550 2077 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:55.845698 2249 ts_tablet_manager.cc:902] T 5abcdc783c6144f28b9f8a9e1e162006 P 4665a3377574417ba0d3165c236533c0: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:55.847937 2159 ts_tablet_manager.cc:1888] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:55.857486 31070 raft_consensus.cc:2932] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:55.857928 790 raft_consensus.cc:1261] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:55.858858 1631 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:55.860704 393 raft_consensus.cc:1261] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:55.863708 30791 catalog_manager.cc:5484] T 0f45824b1fff4302abd03c1fb48c6c7b P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:55.866400 2090 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:55.869879 31180 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f45824b1fff4302abd03c1fb48c6c7b with delete_type TABLET_DATA_TOMBSTONED (TS e7badbe63a8a4d9ca0c78818c6f4c8ce not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45854
I20240503 14:13:55.869750 30796 catalog_manager.cc:5484] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:55.871410 1631 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.872570 393 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.876978 30938 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:55.858690 31326 raft_consensus.cc:1261] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:55.866986 31331 consensus_queue.cc:237] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:13:55.879593 30920 tablet_service.cc:1514] Processing DeleteTablet for tablet 63b2120788a54af78f02449f46bf23c7 with delete_type TABLET_DATA_TOMBSTONED (TS 0dc902d91ccc435a82a5ede4a7729811 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:56642
I20240503 14:13:55.875217 1578 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:55.880360 2155 tablet_replica.cc:330] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811: stopping tablet replica
I20240503 14:13:55.880631 2155 raft_consensus.cc:2229] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:55.881071 2155 raft_consensus.cc:2258] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.881966 2090 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:55.884411 2159 log.cc:1196] T c837def6d76242f08b20a448541e64d6 P 82cfdc130877418dbe9d980d368973be: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/wals/c837def6d76242f08b20a448541e64d6
I20240503 14:13:55.884538 2155 ts_tablet_manager.cc:1875] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:55.886144 1919 tablet_copy_client.cc:783] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:55.886366 1919 tablet_copy_client.cc:647] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:55.887727 30786 catalog_manager.cc:4830] TS 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063): tablet c837def6d76242f08b20a448541e64d6 (table rebalance_test_table_6 [id=08c2ccf24fcc40f9a0fd947d52b50f8e]) successfully deleted
I20240503 14:13:55.888106 1919 tablet_copy_client.cc:523] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:55.892758 31201 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:55.892932 790 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.897130 2260 tablet_replica.cc:330] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce: stopping tablet replica
I20240503 14:13:55.897462 2260 raft_consensus.cc:2229] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:55.897898 2260 raft_consensus.cc:2258] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.900784 30785 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1bb75d7d9499475ca60c633c0801d639 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:55.897576 31326 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.901561 30796 catalog_manager.cc:5484] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5 reported cstate change: config changed from index 7 to 8, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.902904 2260 ts_tablet_manager.cc:1875] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:55.905788 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bb75d7d9499475ca60c633c0801d639 with delete_type TABLET_DATA_TOMBSTONED (TS 6dc4948ed7c740ab89b9e349d151b0e5 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:37226
I20240503 14:13:55.906167 2155 ts_tablet_manager.cc:1888] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 14:13:55.906311 2155 log.cc:1196] T 63b2120788a54af78f02449f46bf23c7 P 0dc902d91ccc435a82a5ede4a7729811: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/63b2120788a54af78f02449f46bf23c7
I20240503 14:13:55.907102 30784 catalog_manager.cc:4830] TS 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): tablet 63b2120788a54af78f02449f46bf23c7 (table rebalance_test_table_0 [id=fa8e781fa3e140b6865ce5f7b429e4c4]) successfully deleted
I20240503 14:13:55.908747 31200 raft_consensus.cc:1261] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:55.909457 1578 consensus_queue.cc:1035] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:55.825196 30800 catalog_manager.cc:5484] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 reported cstate change: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0dc902d91ccc435a82a5ede4a7729811" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.914757 31071 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.917411 2249 tablet_copy_client.cc:308] T 5abcdc783c6144f28b9f8a9e1e162006 P 4665a3377574417ba0d3165c236533c0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:13:55.918172 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet 5abcdc783c6144f28b9f8a9e1e162006 from peer 4665a3377574417ba0d3165c236533c0 ({username='slave'} at 127.22.242.71:41641)
I20240503 14:13:55.918370 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet 5abcdc783c6144f28b9f8a9e1e162006 from peer 4665a3377574417ba0d3165c236533c0 at {username='slave'} at 127.22.242.71:41641: session id = 4665a3377574417ba0d3165c236533c0-5abcdc783c6144f28b9f8a9e1e162006
I20240503 14:13:55.921226 31092 tablet_copy_source_session.cc:205] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:55.922821 2249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5abcdc783c6144f28b9f8a9e1e162006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.923295 2126 raft_consensus.cc:2932] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.927381 30786 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 69b741aad35d4bfc8fea5b8e21d4a733 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:55.928494 31201 raft_consensus.cc:2932] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.932698 660 raft_consensus.cc:2932] T 0f45824b1fff4302abd03c1fb48c6c7b P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:55.933578 659 raft_consensus.cc:1261] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.935845 790 raft_consensus.cc:1261] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:55.943488 2070 tablet_bootstrap.cc:492] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:55.946873 1919 tablet_bootstrap.cc:492] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap starting.
I20240503 14:13:55.947499 2070 tablet_bootstrap.cc:492] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1: Bootstrap complete.
I20240503 14:13:55.948073 30940 consensus_queue.cc:237] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:13:55.950675 2070 ts_tablet_manager.cc:1367] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1: Time spent bootstrapping tablet: real 0.834s user 0.058s sys 0.001s
I20240503 14:13:55.950734 527 raft_consensus.cc:1261] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:55.952692 1914 consensus_queue.cc:1035] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:55.953008 2249 tablet_copy_client.cc:783] T 5abcdc783c6144f28b9f8a9e1e162006 P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:55.953311 2249 tablet_copy_client.cc:647] T 5abcdc783c6144f28b9f8a9e1e162006 P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:55.953996 2265 tablet_replica.cc:330] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5: stopping tablet replica
I20240503 14:13:55.954571 2265 raft_consensus.cc:2229] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:55.955035 2265 raft_consensus.cc:2258] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.955698 1914 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.956398 30791 catalog_manager.cc:5484] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:55.958825 31202 raft_consensus.cc:1261] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:55.959566 2143 consensus_queue.cc:1035] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:55.959779 30784 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1a9b84d307b8432bbdf6b70a11bf66b9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:55.960680 528 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.961014 2260 ts_tablet_manager.cc:1888] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:55.961531 2260 log.cc:1196] T 0f45824b1fff4302abd03c1fb48c6c7b P e7badbe63a8a4d9ca0c78818c6f4c8ce: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/0f45824b1fff4302abd03c1fb48c6c7b
I20240503 14:13:55.961248 31199 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.962339 2070 raft_consensus.cc:348] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:55.962627 2070 raft_consensus.cc:729] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6967bdda50404d68a5ac043b709440e1, State: Initialized, Role: LEARNER
I20240503 14:13:55.962839 2070 consensus_queue.cc:260] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:55.964466 31198 consensus_queue.cc:237] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:13:55.966681 2070 ts_tablet_manager.cc:1398] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240503 14:13:55.967101 393 raft_consensus.cc:1261] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:13:55.967715 790 raft_consensus.cc:1261] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:13:55.967775 2090 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:55.968533 31091 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session 6967bdda50404d68a5ac043b709440e1-c86ee72320b647eb84e376f8995b55d2 received from {username='slave'} at 127.22.242.69:44021
I20240503 14:13:55.968843 1628 consensus_queue.cc:1035] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:55.970549 30785 catalog_manager.cc:4830] TS e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297): tablet 0f45824b1fff4302abd03c1fb48c6c7b (table rebalance_test_table_7 [id=5b67f0a6a8d34286994597065f21a92b]) successfully deleted
I20240503 14:13:55.971134 31091 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session 6967bdda50404d68a5ac043b709440e1-c86ee72320b647eb84e376f8995b55d2 on tablet c86ee72320b647eb84e376f8995b55d2 with peer 6967bdda50404d68a5ac043b709440e1
I20240503 14:13:55.971866 30791 catalog_manager.cc:5484] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 reported cstate change: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "0dc902d91ccc435a82a5ede4a7729811" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:55.972772 2090 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.973495 2249 tablet_copy_client.cc:523] T 5abcdc783c6144f28b9f8a9e1e162006 P 4665a3377574417ba0d3165c236533c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:55.976305 790 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.976344 393 raft_consensus.cc:2932] T 1bb75d7d9499475ca60c633c0801d639 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:55.983660 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:55.989471 2249 tablet_bootstrap.cc:492] T 5abcdc783c6144f28b9f8a9e1e162006 P 4665a3377574417ba0d3165c236533c0: Bootstrap starting.
I20240503 14:13:55.992931 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a9b84d307b8432bbdf6b70a11bf66b9 with delete_type TABLET_DATA_TOMBSTONED (TS 6dc4948ed7c740ab89b9e349d151b0e5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37226
I20240503 14:13:55.993764 2265 ts_tablet_manager.cc:1875] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:56.007086 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:56.007834 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:56.007825 30797 catalog_manager.cc:5484] T 1bb75d7d9499475ca60c633c0801d639 P 85f69eefe4294197923dff5efadcbbd5 reported cstate change: config changed from index 8 to 9, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:56.026504 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:56.030405 2070 ts_tablet_manager.cc:902] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1: Initiating tablet copy from peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:56.031630 31311 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bb75d7d9499475ca60c633c0801d639 with delete_type TABLET_DATA_TOMBSTONED (TS 82cfdc130877418dbe9d980d368973be not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48102
I20240503 14:13:56.031989 2159 tablet_replica.cc:330] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be: stopping tablet replica
I20240503 14:13:56.032234 2159 raft_consensus.cc:2229] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:56.032620 2159 raft_consensus.cc:2258] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:13:56.036080 31268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.66:33221, user_credentials={real_user=slave}} blocked reactor thread for 56419.8us
I20240503 14:13:56.036897 2159 ts_tablet_manager.cc:1875] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:56.039454 31331 consensus_queue.cc:237] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:56.049197 2077 consensus_queue.cc:1035] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:56.057273 2077 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:56.060768 2159 ts_tablet_manager.cc:1888] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:13:56.060783 790 raft_consensus.cc:2932] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:56.061264 30797 catalog_manager.cc:5484] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:56.062314 507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:56.079337 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:56.101902 2159 log.cc:1196] T 1bb75d7d9499475ca60c633c0801d639 P 82cfdc130877418dbe9d980d368973be: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/wals/1bb75d7d9499475ca60c633c0801d639
I20240503 14:13:56.107583 30785 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1bb75d7d9499475ca60c633c0801d639 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:56.108649 1628 raft_consensus.cc:1050] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce: attempting to promote NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 to VOTER
I20240503 14:13:56.109437 1628 consensus_queue.cc:237] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:13:56.110744 660 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
W20240503 14:13:56.111361 31265 consensus_peers.cc:485] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be -> Peer fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829): Couldn't send request to peer fa71cbdf0e544b81bd4e7e27196d2c91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23476c4bcf6e4b4085d59d29739b6edb. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:56.112468 31072 raft_consensus.cc:1261] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:56.113090 2077 consensus_queue.cc:1035] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:56.114075 31199 raft_consensus.cc:1261] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:56.114599 2077 consensus_queue.cc:1035] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:56.120826 30786 catalog_manager.cc:4830] TS 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063): tablet 1bb75d7d9499475ca60c633c0801d639 (table rebalance_test_table_1 [id=f6d48b36155b41d3a75fe51d7d204ea7]) successfully deleted
I20240503 14:13:56.123061 2070 tablet_copy_client.cc:308] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:46013
I20240503 14:13:56.137363 393 raft_consensus.cc:1203] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:56.145891 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:56.159739 2154 tablet_bootstrap.cc:492] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:56.160283 2154 tablet_bootstrap.cc:492] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap complete.
I20240503 14:13:56.160570 2154 ts_tablet_manager.cc:1367] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5: Time spent bootstrapping tablet: real 0.446s user 0.045s sys 0.008s
I20240503 14:13:56.161360 2154 raft_consensus.cc:348] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:56.161664 2154 raft_consensus.cc:729] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85f69eefe4294197923dff5efadcbbd5, State: Initialized, Role: LEARNER
I20240503 14:13:56.161883 2154 consensus_queue.cc:260] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:56.162425 2154 ts_tablet_manager.cc:1398] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:56.163185 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session 85f69eefe4294197923dff5efadcbbd5-f3e80559017d4a6ba16d34e0b7aab085 received from {username='slave'} at 127.22.242.72:54703
I20240503 14:13:56.163311 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session 85f69eefe4294197923dff5efadcbbd5-f3e80559017d4a6ba16d34e0b7aab085 on tablet f3e80559017d4a6ba16d34e0b7aab085 with peer 85f69eefe4294197923dff5efadcbbd5
I20240503 14:13:56.179528 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet 69b741aad35d4bfc8fea5b8e21d4a733 with delete_type TABLET_DATA_TOMBSTONED (TS 6dc4948ed7c740ab89b9e349d151b0e5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37226
I20240503 14:13:56.180049 31311 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bb75d7d9499475ca60c633c0801d639 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 8 (current committed config index is 9)) from {username='slave'} at 127.0.0.1:48102
I20240503 14:13:56.180624 30961 tablet_copy_service.cc:140] P 0dc902d91ccc435a82a5ede4a7729811: Received BeginTabletCopySession request for tablet f6aeaa3fe1cd4127a6144d85b0434057 from peer 6967bdda50404d68a5ac043b709440e1 ({username='slave'} at 127.22.242.69:60815)
I20240503 14:13:56.180846 30961 tablet_copy_service.cc:161] P 0dc902d91ccc435a82a5ede4a7729811: Beginning new tablet copy session on tablet f6aeaa3fe1cd4127a6144d85b0434057 from peer 6967bdda50404d68a5ac043b709440e1 at {username='slave'} at 127.22.242.69:60815: session id = 6967bdda50404d68a5ac043b709440e1-f6aeaa3fe1cd4127a6144d85b0434057
I20240503 14:13:56.181195 30786 catalog_manager.cc:4830] TS 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063): tablet 1bb75d7d9499475ca60c633c0801d639 (table rebalance_test_table_1 [id=f6d48b36155b41d3a75fe51d7d204ea7]) successfully deleted
I20240503 14:13:56.182363 2265 ts_tablet_manager.cc:1888] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 14:13:56.182574 2265 log.cc:1196] T 1bb75d7d9499475ca60c633c0801d639 P 6dc4948ed7c740ab89b9e349d151b0e5: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/1bb75d7d9499475ca60c633c0801d639
I20240503 14:13:56.184010 30961 tablet_copy_source_session.cc:205] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:56.184294 30787 catalog_manager.cc:4830] TS 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221): tablet 1bb75d7d9499475ca60c633c0801d639 (table rebalance_test_table_1 [id=f6d48b36155b41d3a75fe51d7d204ea7]) successfully deleted
I20240503 14:13:56.184365 2265 tablet_replica.cc:330] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5: stopping tablet replica
I20240503 14:13:56.185900 2070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6aeaa3fe1cd4127a6144d85b0434057. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:56.193113 2070 tablet_copy_client.cc:783] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:56.193497 2070 tablet_copy_client.cc:647] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:56.195330 2070 tablet_copy_client.cc:523] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:56.205984 2070 tablet_bootstrap.cc:492] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1: Bootstrap starting.
I20240503 14:13:56.206243 1919 tablet_bootstrap.cc:492] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:56.206696 1919 tablet_bootstrap.cc:492] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap complete.
I20240503 14:13:56.184600 2265 raft_consensus.cc:2229] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:56.216732 2265 raft_consensus.cc:2258] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:56.217830 2265 ts_tablet_manager.cc:1875] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:56.222307 1919 ts_tablet_manager.cc:1367] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent bootstrapping tablet: real 0.276s user 0.057s sys 0.001s
I20240503 14:13:56.223261 1919 raft_consensus.cc:348] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:56.223548 1919 raft_consensus.cc:729] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa71cbdf0e544b81bd4e7e27196d2c91, State: Initialized, Role: LEARNER
I20240503 14:13:56.223809 1919 consensus_queue.cc:260] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:56.225178 1919 ts_tablet_manager.cc:1398] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:56.226429 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-584769402a7e4e8a985a8f2363e2b8b9 received from {username='slave'} at 127.22.242.70:40181
I20240503 14:13:56.226567 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-584769402a7e4e8a985a8f2363e2b8b9 on tablet 584769402a7e4e8a985a8f2363e2b8b9 with peer fa71cbdf0e544b81bd4e7e27196d2c91
I20240503 14:13:56.241479 2126 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:56.241708 31201 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:56.245383 31331 raft_consensus.cc:1261] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:56.247054 1628 consensus_queue.cc:1035] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:56.252197 31072 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:56.257553 30794 catalog_manager.cc:5484] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:13:56.272953 31135 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.65:46013, user_credentials={real_user=slave}} blocked reactor thread for 174421us
I20240503 14:13:56.277284 30938 raft_consensus.cc:1261] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:56.283704 2265 ts_tablet_manager.cc:1888] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:56.289769 2280 consensus_queue.cc:1035] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:56.308871 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:56.324420 527 raft_consensus.cc:1261] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:56.326551 2070 tablet_bootstrap.cc:492] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:56.327085 2070 tablet_bootstrap.cc:492] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1: Bootstrap complete.
I20240503 14:13:56.327335 2070 ts_tablet_manager.cc:1367] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1: Time spent bootstrapping tablet: real 0.121s user 0.057s sys 0.000s
I20240503 14:13:56.328226 2070 raft_consensus.cc:348] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:56.328451 2070 raft_consensus.cc:729] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6967bdda50404d68a5ac043b709440e1, State: Initialized, Role: LEARNER
I20240503 14:13:56.328534 1919 ts_tablet_manager.cc:902] T dd5869fd8f8d41d99cf531edb7df3750 P fa71cbdf0e544b81bd4e7e27196d2c91: Initiating tablet copy from peer e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:56.328639 2070 consensus_queue.cc:260] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:56.330073 2070 ts_tablet_manager.cc:1398] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:56.330785 30961 tablet_copy_service.cc:342] P 0dc902d91ccc435a82a5ede4a7729811: Request end of tablet copy session 6967bdda50404d68a5ac043b709440e1-f6aeaa3fe1cd4127a6144d85b0434057 received from {username='slave'} at 127.22.242.69:60815
I20240503 14:13:56.330912 30961 tablet_copy_service.cc:433] P 0dc902d91ccc435a82a5ede4a7729811: ending tablet copy session 6967bdda50404d68a5ac043b709440e1-f6aeaa3fe1cd4127a6144d85b0434057 on tablet f6aeaa3fe1cd4127a6144d85b0434057 with peer 6967bdda50404d68a5ac043b709440e1
I20240503 14:13:56.335175 1919 tablet_copy_client.cc:308] T dd5869fd8f8d41d99cf531edb7df3750 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44297
I20240503 14:13:56.340682 31221 tablet_copy_service.cc:140] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Received BeginTabletCopySession request for tablet dd5869fd8f8d41d99cf531edb7df3750 from peer fa71cbdf0e544b81bd4e7e27196d2c91 ({username='slave'} at 127.22.242.70:33209)
I20240503 14:13:56.340941 31221 tablet_copy_service.cc:161] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Beginning new tablet copy session on tablet dd5869fd8f8d41d99cf531edb7df3750 from peer fa71cbdf0e544b81bd4e7e27196d2c91 at {username='slave'} at 127.22.242.70:33209: session id = fa71cbdf0e544b81bd4e7e27196d2c91-dd5869fd8f8d41d99cf531edb7df3750
I20240503 14:13:56.344421 31221 tablet_copy_source_session.cc:205] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:56.352051 2280 raft_consensus.cc:2932] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:56.358959 30938 raft_consensus.cc:2932] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:56.360782 1919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5869fd8f8d41d99cf531edb7df3750. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:56.368378 2090 consensus_queue.cc:1035] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:56.373039 2265 log.cc:1196] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6dc4948ed7c740ab89b9e349d151b0e5: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/1a9b84d307b8432bbdf6b70a11bf66b9
I20240503 14:13:56.374433 2265 tablet_replica.cc:330] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5: stopping tablet replica
I20240503 14:13:56.374704 2265 raft_consensus.cc:2229] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:56.374989 2265 raft_consensus.cc:2258] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:56.376308 2265 ts_tablet_manager.cc:1875] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:56.384678 2249 tablet_bootstrap.cc:492] T 5abcdc783c6144f28b9f8a9e1e162006 P 4665a3377574417ba0d3165c236533c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:56.385219 2249 tablet_bootstrap.cc:492] T 5abcdc783c6144f28b9f8a9e1e162006 P 4665a3377574417ba0d3165c236533c0: Bootstrap complete.
I20240503 14:13:56.378131 31331 raft_consensus.cc:2932] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:56.385522 2249 ts_tablet_manager.cc:1367] T 5abcdc783c6144f28b9f8a9e1e162006 P 4665a3377574417ba0d3165c236533c0: Time spent bootstrapping tablet: real 0.396s user 0.044s sys 0.008s
I20240503 14:13:56.386390 2249 raft_consensus.cc:348] T 5abcdc783c6144f28b9f8a9e1e162006 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:56.386668 2249 raft_consensus.cc:729] T 5abcdc783c6144f28b9f8a9e1e162006 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4665a3377574417ba0d3165c236533c0, State: Initialized, Role: LEARNER
I20240503 14:13:56.386909 2249 consensus_queue.cc:260] T 5abcdc783c6144f28b9f8a9e1e162006 P 4665a3377574417ba0d3165c236533c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:56.388466 2249 ts_tablet_manager.cc:1398] T 5abcdc783c6144f28b9f8a9e1e162006 P 4665a3377574417ba0d3165c236533c0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:56.389128 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session 4665a3377574417ba0d3165c236533c0-5abcdc783c6144f28b9f8a9e1e162006 received from {username='slave'} at 127.22.242.71:41641
I20240503 14:13:56.389262 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session 4665a3377574417ba0d3165c236533c0-5abcdc783c6144f28b9f8a9e1e162006 on tablet 5abcdc783c6144f28b9f8a9e1e162006 with peer 4665a3377574417ba0d3165c236533c0
I20240503 14:13:56.399708 528 raft_consensus.cc:2932] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:56.412268 30787 catalog_manager.cc:4830] TS 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221): tablet 1a9b84d307b8432bbdf6b70a11bf66b9 (table rebalance_test_table_8 [id=c243ced42cda4e3996901964941b50f4]) successfully deleted
I20240503 14:13:56.416018 30793 catalog_manager.cc:5484] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:56.429489 31072 consensus_queue.cc:237] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:56.442070 1919 tablet_copy_client.cc:783] T dd5869fd8f8d41d99cf531edb7df3750 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:56.442346 1919 tablet_copy_client.cc:647] T dd5869fd8f8d41d99cf531edb7df3750 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 1 WAL segments...
W20240503 14:13:56.446815 31008 consensus_peers.cc:485] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 589fce6d925b43d895588bac355135b0. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:56.448135 30938 raft_consensus.cc:1261] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:56.449023 2069 consensus_queue.cc:1035] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:56.451004 31331 raft_consensus.cc:1261] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:56.451996 2069 consensus_queue.cc:1035] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:56.459280 2265 ts_tablet_manager.cc:1888] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:56.462352 30940 raft_consensus.cc:2932] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:56.463817 1919 tablet_copy_client.cc:523] T dd5869fd8f8d41d99cf531edb7df3750 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:56.470633 30798 catalog_manager.cc:5484] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:56.472720 1851 tablet_bootstrap.cc:492] T 6234489254fc4882a39b7df28ddf673f P 4665a3377574417ba0d3165c236533c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:56.476503 1851 tablet_bootstrap.cc:492] T 6234489254fc4882a39b7df28ddf673f P 4665a3377574417ba0d3165c236533c0: Bootstrap complete.
I20240503 14:13:56.476861 1851 ts_tablet_manager.cc:1367] T 6234489254fc4882a39b7df28ddf673f P 4665a3377574417ba0d3165c236533c0: Time spent bootstrapping tablet: real 1.118s user 0.052s sys 0.007s
I20240503 14:13:56.477828 1851 raft_consensus.cc:348] T 6234489254fc4882a39b7df28ddf673f P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:56.478094 1851 raft_consensus.cc:729] T 6234489254fc4882a39b7df28ddf673f P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4665a3377574417ba0d3165c236533c0, State: Initialized, Role: LEARNER
I20240503 14:13:56.478317 1851 consensus_queue.cc:260] T 6234489254fc4882a39b7df28ddf673f P 4665a3377574417ba0d3165c236533c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:56.478957 1851 ts_tablet_manager.cc:1398] T 6234489254fc4882a39b7df28ddf673f P 4665a3377574417ba0d3165c236533c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:56.479794 31352 tablet_copy_service.cc:342] P 82cfdc130877418dbe9d980d368973be: Request end of tablet copy session 4665a3377574417ba0d3165c236533c0-6234489254fc4882a39b7df28ddf673f received from {username='slave'} at 127.22.242.71:44595
I20240503 14:13:56.479939 31352 tablet_copy_service.cc:433] P 82cfdc130877418dbe9d980d368973be: ending tablet copy session 4665a3377574417ba0d3165c236533c0-6234489254fc4882a39b7df28ddf673f on tablet 6234489254fc4882a39b7df28ddf673f with peer 4665a3377574417ba0d3165c236533c0
I20240503 14:13:56.461586 1600 raft_consensus.cc:2932] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:56.484721 2265 log.cc:1196] T 69b741aad35d4bfc8fea5b8e21d4a733 P 6dc4948ed7c740ab89b9e349d151b0e5: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/69b741aad35d4bfc8fea5b8e21d4a733
I20240503 14:13:56.488444 30787 catalog_manager.cc:4830] TS 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221): tablet 69b741aad35d4bfc8fea5b8e21d4a733 (table rebalance_test_table_8 [id=c243ced42cda4e3996901964941b50f4]) successfully deleted
I20240503 14:13:56.488956 1919 tablet_bootstrap.cc:492] T dd5869fd8f8d41d99cf531edb7df3750 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap starting.
I20240503 14:13:56.491092 31326 raft_consensus.cc:2932] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:56.515228 660 raft_consensus.cc:1203] T 6234489254fc4882a39b7df28ddf673f P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:56.527822 2191 tablet_bootstrap.cc:492] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:56.528470 2191 tablet_bootstrap.cc:492] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0: Bootstrap complete.
I20240503 14:13:56.528825 2191 ts_tablet_manager.cc:1367] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0: Time spent bootstrapping tablet: real 0.767s user 0.049s sys 0.009s
I20240503 14:13:56.529772 2191 raft_consensus.cc:348] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:56.530100 2191 raft_consensus.cc:729] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4665a3377574417ba0d3165c236533c0, State: Initialized, Role: LEARNER
I20240503 14:13:56.530333 2191 consensus_queue.cc:260] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:56.530938 2191 ts_tablet_manager.cc:1398] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:56.531718 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session 4665a3377574417ba0d3165c236533c0-ebd6dc710e144e61a5fae3911f25ee5c received from {username='slave'} at 127.22.242.71:41641
I20240503 14:13:56.531852 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session 4665a3377574417ba0d3165c236533c0-ebd6dc710e144e61a5fae3911f25ee5c on tablet ebd6dc710e144e61a5fae3911f25ee5c with peer 4665a3377574417ba0d3165c236533c0
I20240503 14:13:56.562161 393 raft_consensus.cc:1203] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:56.564716 660 raft_consensus.cc:1203] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:56.566059 2326 ts_tablet_manager.cc:902] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91: Initiating tablet copy from peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:56.599120 31326 consensus_queue.cc:237] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:56.602880 31201 raft_consensus.cc:1261] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:56.603588 30938 raft_consensus.cc:1261] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:56.604615 2126 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:56.605823 2126 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:56.613179 2126 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:56.619000 31201 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:56.640081 30938 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
W20240503 14:13:56.640998 30786 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet bc77a7da906b419085258f72667ff185 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 5 but the committed config has opid_index of 6
I20240503 14:13:56.650528 30798 catalog_manager.cc:5484] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 5 to 6, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:56.670895 790 raft_consensus.cc:1203] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:56.693934 2069 raft_consensus.cc:1050] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER 6967bdda50404d68a5ac043b709440e1 to VOTER
I20240503 14:13:56.696422 660 raft_consensus.cc:1261] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:56.699904 1919 tablet_bootstrap.cc:492] T dd5869fd8f8d41d99cf531edb7df3750 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:56.704092 1919 tablet_bootstrap.cc:492] T dd5869fd8f8d41d99cf531edb7df3750 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap complete.
I20240503 14:13:56.704423 1919 ts_tablet_manager.cc:1367] T dd5869fd8f8d41d99cf531edb7df3750 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent bootstrapping tablet: real 0.216s user 0.057s sys 0.013s
I20240503 14:13:56.705224 1919 raft_consensus.cc:348] T dd5869fd8f8d41d99cf531edb7df3750 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:56.705502 1919 raft_consensus.cc:729] T dd5869fd8f8d41d99cf531edb7df3750 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa71cbdf0e544b81bd4e7e27196d2c91, State: Initialized, Role: LEARNER
I20240503 14:13:56.705713 1919 consensus_queue.cc:260] T dd5869fd8f8d41d99cf531edb7df3750 P fa71cbdf0e544b81bd4e7e27196d2c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:56.706290 1919 ts_tablet_manager.cc:1398] T dd5869fd8f8d41d99cf531edb7df3750 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:56.707643 31221 tablet_copy_service.cc:342] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Request end of tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-dd5869fd8f8d41d99cf531edb7df3750 received from {username='slave'} at 127.22.242.70:33209
I20240503 14:13:56.707794 31221 tablet_copy_service.cc:433] P e7badbe63a8a4d9ca0c78818c6f4c8ce: ending tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-dd5869fd8f8d41d99cf531edb7df3750 on tablet dd5869fd8f8d41d99cf531edb7df3750 with peer fa71cbdf0e544b81bd4e7e27196d2c91
I20240503 14:13:56.694880 2069 consensus_queue.cc:237] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:13:56.711565 2326 tablet_copy_client.cc:308] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.68:37063
I20240503 14:13:56.712813 31352 tablet_copy_service.cc:140] P 82cfdc130877418dbe9d980d368973be: Received BeginTabletCopySession request for tablet 23476c4bcf6e4b4085d59d29739b6edb from peer fa71cbdf0e544b81bd4e7e27196d2c91 ({username='slave'} at 127.22.242.70:37791)
I20240503 14:13:56.713001 31352 tablet_copy_service.cc:161] P 82cfdc130877418dbe9d980d368973be: Beginning new tablet copy session on tablet 23476c4bcf6e4b4085d59d29739b6edb from peer fa71cbdf0e544b81bd4e7e27196d2c91 at {username='slave'} at 127.22.242.70:37791: session id = fa71cbdf0e544b81bd4e7e27196d2c91-23476c4bcf6e4b4085d59d29739b6edb
I20240503 14:13:56.713728 660 raft_consensus.cc:1203] T 5abcdc783c6144f28b9f8a9e1e162006 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:56.715816 31352 tablet_copy_source_session.cc:205] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:56.720135 2326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23476c4bcf6e4b4085d59d29739b6edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:56.736276 2326 tablet_copy_client.cc:783] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:56.736726 2326 tablet_copy_client.cc:647] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:56.740552 2126 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:56.744084 31326 raft_consensus.cc:1261] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:56.744802 393 raft_consensus.cc:1261] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:56.745465 2069 consensus_queue.cc:1035] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:56.745784 2326 tablet_copy_client.cc:523] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:56.745839 2079 consensus_queue.cc:1035] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:56.746170 31201 raft_consensus.cc:1261] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:56.746750 2079 consensus_queue.cc:1035] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:56.751272 528 raft_consensus.cc:1203] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:56.762338 2326 tablet_bootstrap.cc:492] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap starting.
I20240503 14:13:56.783190 2069 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:56.790022 393 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:56.793040 31201 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:56.802520 31326 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:56.802508 660 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:56.814145 30797 catalog_manager.cc:5484] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:56.843225 31070 consensus_queue.cc:237] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:56.853886 2326 tablet_bootstrap.cc:492] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:56.854486 2326 tablet_bootstrap.cc:492] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap complete.
I20240503 14:13:56.854813 2326 ts_tablet_manager.cc:1367] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent bootstrapping tablet: real 0.093s user 0.056s sys 0.014s
I20240503 14:13:56.855643 2326 raft_consensus.cc:348] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:56.856333 2326 raft_consensus.cc:729] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa71cbdf0e544b81bd4e7e27196d2c91, State: Initialized, Role: LEARNER
I20240503 14:13:56.856612 2326 consensus_queue.cc:260] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:56.858108 2326 ts_tablet_manager.cc:1398] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:56.858758 31352 tablet_copy_service.cc:342] P 82cfdc130877418dbe9d980d368973be: Request end of tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-23476c4bcf6e4b4085d59d29739b6edb received from {username='slave'} at 127.22.242.70:37791
I20240503 14:13:56.858884 31352 tablet_copy_service.cc:433] P 82cfdc130877418dbe9d980d368973be: ending tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-23476c4bcf6e4b4085d59d29739b6edb on tablet 23476c4bcf6e4b4085d59d29739b6edb with peer fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:56.865069 31268 consensus_peers.cc:485] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be -> Peer 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189): Couldn't send request to peer 6967bdda50404d68a5ac043b709440e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc77a7da906b419085258f72667ff185. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:56.866632 31072 consensus_queue.cc:237] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:13:56.868487 30938 raft_consensus.cc:1261] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:56.869127 31326 raft_consensus.cc:1261] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:56.869364 393 raft_consensus.cc:1261] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:56.869799 2069 consensus_queue.cc:1035] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:56.870141 1602 consensus_queue.cc:1035] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:56.870335 2079 consensus_queue.cc:1035] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:13:56.870844 31008 consensus_peers.cc:485] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ff2fa69f79644c5b93beeba32fc9ca0. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:56.872958 2069 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:56.874293 393 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:56.878293 30787 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c86ee72320b647eb84e376f8995b55d2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:56.883172 31200 raft_consensus.cc:1261] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:56.884531 2069 consensus_queue.cc:1035] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:56.886145 31200 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:56.891124 30794 catalog_manager.cc:5484] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:56.892243 528 raft_consensus.cc:1203] T dd5869fd8f8d41d99cf531edb7df3750 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:56.893371 2069 raft_consensus.cc:2932] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:56.896397 31311 tablet_service.cc:1514] Processing DeleteTablet for tablet c86ee72320b647eb84e376f8995b55d2 with delete_type TABLET_DATA_TOMBSTONED (TS 82cfdc130877418dbe9d980d368973be not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48102
I20240503 14:13:56.896914 31331 raft_consensus.cc:2932] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:56.904079 30797 catalog_manager.cc:5484] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:56.929740 2395 tablet_replica.cc:330] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be: stopping tablet replica
I20240503 14:13:56.930128 2395 raft_consensus.cc:2229] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:56.930539 2395 raft_consensus.cc:2258] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:56.932196 2395 ts_tablet_manager.cc:1875] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:56.943195 1722 raft_consensus.cc:1050] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be: attempting to promote NON_VOTER 4665a3377574417ba0d3165c236533c0 to VOTER
I20240503 14:13:56.944123 1722 consensus_queue.cc:237] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:13:56.946976 30940 raft_consensus.cc:1261] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:56.947340 31070 raft_consensus.cc:1261] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:56.947443 660 raft_consensus.cc:1261] T 6234489254fc4882a39b7df28ddf673f P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:56.947983 1850 consensus_queue.cc:1035] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:56.948347 1722 consensus_queue.cc:1035] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:56.948707 1578 consensus_queue.cc:1035] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:56.963326 2395 ts_tablet_manager.cc:1888] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:56.966240 528 raft_consensus.cc:1203] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:56.970575 30936 raft_consensus.cc:2932] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:56.971388 659 raft_consensus.cc:2932] T 6234489254fc4882a39b7df28ddf673f P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:56.970582 30939 raft_consensus.cc:2932] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:56.975409 30794 catalog_manager.cc:5484] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:56.988710 31200 consensus_queue.cc:237] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:13:56.991182 528 raft_consensus.cc:1261] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:56.991979 2090 consensus_queue.cc:1035] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:56.995023 2090 raft_consensus.cc:2932] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:56.999933 528 raft_consensus.cc:2932] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.008569 30785 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 91a7b8a53140429b956178d43478a541 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:57.011726 2143 raft_consensus.cc:1050] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811: attempting to promote NON_VOTER 6967bdda50404d68a5ac043b709440e1 to VOTER
I20240503 14:13:57.012696 2143 consensus_queue.cc:237] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:13:57.014868 30939 raft_consensus.cc:1261] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:57.015478 2294 consensus_queue.cc:1035] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:57.017750 30939 raft_consensus.cc:2932] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.023257 30793 catalog_manager.cc:5484] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:57.028892 2406 ts_tablet_manager.cc:902] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: Initiating tablet copy from peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:57.030666 394 raft_consensus.cc:1261] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.031540 1914 consensus_queue.cc:1035] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.040647 31070 raft_consensus.cc:1261] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.041580 2143 consensus_queue.cc:1035] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 14:13:56.969233 1578 raft_consensus.cc:2932] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.033329 31326 raft_consensus.cc:1261] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.036201 31311 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a7b8a53140429b956178d43478a541 with delete_type TABLET_DATA_TOMBSTONED (TS 82cfdc130877418dbe9d980d368973be not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48102
I20240503 14:13:57.056713 1914 consensus_queue.cc:1035] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.057870 1914 raft_consensus.cc:2932] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:57.069301 2069 raft_consensus.cc:1050] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER 85f69eefe4294197923dff5efadcbbd5 to VOTER
I20240503 14:13:57.069808 31070 raft_consensus.cc:2932] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:57.070315 2069 consensus_queue.cc:237] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:13:57.073006 792 raft_consensus.cc:1261] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.074002 1602 consensus_queue.cc:1035] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.074592 31072 raft_consensus.cc:2932] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.075090 31331 raft_consensus.cc:1261] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.075724 1602 consensus_queue.cc:1035] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.078797 31201 raft_consensus.cc:1261] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.081468 1602 consensus_queue.cc:1035] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.090037 31328 consensus_queue.cc:237] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:13:57.093780 30939 raft_consensus.cc:1261] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:57.094404 1722 consensus_queue.cc:1035] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:57.096491 1602 raft_consensus.cc:1050] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER 4665a3377574417ba0d3165c236533c0 to VOTER
I20240503 14:13:57.097764 30939 raft_consensus.cc:2932] T 6234489254fc4882a39b7df28ddf673f P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.100045 394 raft_consensus.cc:2932] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:57.103238 1578 raft_consensus.cc:2932] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.106624 30793 catalog_manager.cc:5484] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 reported cstate change: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0dc902d91ccc435a82a5ede4a7729811" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:57.110621 30786 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6234489254fc4882a39b7df28ddf673f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:57.097414 1602 consensus_queue.cc:237] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:13:57.113056 30794 catalog_manager.cc:5484] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:57.115895 30939 consensus_queue.cc:237] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:13:57.117431 660 raft_consensus.cc:1261] T 6234489254fc4882a39b7df28ddf673f P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:57.117686 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet 6234489254fc4882a39b7df28ddf673f with delete_type TABLET_DATA_TOMBSTONED (TS 6dc4948ed7c740ab89b9e349d151b0e5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37226
I20240503 14:13:57.119571 31070 raft_consensus.cc:1261] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:57.121341 1914 consensus_queue.cc:1035] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:57.121735 393 raft_consensus.cc:1261] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:57.122251 2138 consensus_queue.cc:1035] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:57.123769 1578 consensus_queue.cc:1035] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:57.127336 1914 raft_consensus.cc:2932] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:57.127951 2395 log.cc:1196] T c86ee72320b647eb84e376f8995b55d2 P 82cfdc130877418dbe9d980d368973be: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/wals/c86ee72320b647eb84e376f8995b55d2
I20240503 14:13:57.128150 660 raft_consensus.cc:2932] T 6234489254fc4882a39b7df28ddf673f P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.133266 2395 tablet_replica.cc:330] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be: stopping tablet replica
I20240503 14:13:57.133330 30786 catalog_manager.cc:4830] TS 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063): tablet c86ee72320b647eb84e376f8995b55d2 (table rebalance_test_table_7 [id=5b67f0a6a8d34286994597065f21a92b]) successfully deleted
I20240503 14:13:57.133505 2395 raft_consensus.cc:2229] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:57.133785 2395 raft_consensus.cc:2258] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:57.135135 394 raft_consensus.cc:2932] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:57.138873 30784 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f6aeaa3fe1cd4127a6144d85b0434057 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:57.141064 31070 raft_consensus.cc:2932] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:57.144325 30797 catalog_manager.cc:5484] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 reported cstate change: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New cstate: current_term: 1 leader_uuid: "0dc902d91ccc435a82a5ede4a7729811" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:57.149037 31311 tablet_service.cc:1514] Processing DeleteTablet for tablet f6aeaa3fe1cd4127a6144d85b0434057 with delete_type TABLET_DATA_TOMBSTONED (TS 82cfdc130877418dbe9d980d368973be not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48102
I20240503 14:13:57.150215 2434 tablet_replica.cc:330] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5: stopping tablet replica
I20240503 14:13:57.151806 2434 raft_consensus.cc:2229] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:57.152840 2434 raft_consensus.cc:2258] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:57.157706 660 raft_consensus.cc:1261] T 5abcdc783c6144f28b9f8a9e1e162006 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.157864 30939 raft_consensus.cc:1261] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.158528 2069 consensus_queue.cc:1035] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.158823 1600 consensus_queue.cc:1035] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.160112 2434 ts_tablet_manager.cc:1875] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:57.161247 1600 raft_consensus.cc:2932] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:57.167577 31199 raft_consensus.cc:1261] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.167855 31201 raft_consensus.cc:2932] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:57.169193 792 raft_consensus.cc:2932] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:57.169791 2069 consensus_queue.cc:1035] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.174947 2079 raft_consensus.cc:1050] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 to VOTER
I20240503 14:13:57.175745 2079 consensus_queue.cc:237] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:13:57.173300 30793 catalog_manager.cc:5484] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:57.180555 2426 ts_tablet_manager.cc:902] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:57.191723 2395 ts_tablet_manager.cc:1875] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:57.192992 30939 consensus_queue.cc:237] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:57.199600 31202 raft_consensus.cc:1261] T 5051c4e81ea14146a9f1d82b7f5b5f1d P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:57.200416 2143 consensus_queue.cc:1035] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:57.223249 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:57.249934 2434 ts_tablet_manager.cc:1888] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:57.258548 2426 tablet_copy_client.cc:308] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:13:57.261979 2395 ts_tablet_manager.cc:1888] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:57.262167 2395 log.cc:1196] T 91a7b8a53140429b956178d43478a541 P 82cfdc130877418dbe9d980d368973be: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/wals/91a7b8a53140429b956178d43478a541
I20240503 14:13:57.265517 2406 tablet_copy_client.cc:308] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.68:37063
I20240503 14:13:57.274677 528 raft_consensus.cc:1261] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:13:57.280647 30877 consensus_peers.cc:485] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5051c4e81ea14146a9f1d82b7f5b5f1d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:57.282680 2434 log.cc:1196] T 6234489254fc4882a39b7df28ddf673f P 6dc4948ed7c740ab89b9e349d151b0e5: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/6234489254fc4882a39b7df28ddf673f
W20240503 14:13:57.284240 31008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.72:36545, user_credentials={real_user=slave}} blocked reactor thread for 102515us
W20240503 14:13:57.271032 31007 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.68:37063, user_credentials={real_user=slave}} blocked reactor thread for 81800us
I20240503 14:13:57.272068 2069 raft_consensus.cc:2932] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.286726 31199 raft_consensus.cc:2932] T 5051c4e81ea14146a9f1d82b7f5b5f1d P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:57.290339 2079 consensus_queue.cc:1035] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.293763 31200 raft_consensus.cc:2932] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.297533 30787 catalog_manager.cc:4830] TS 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221): tablet 6234489254fc4882a39b7df28ddf673f (table rebalance_test_table_6 [id=08c2ccf24fcc40f9a0fd947d52b50f8e]) successfully deleted
I20240503 14:13:57.301223 30794 catalog_manager.cc:5484] T 5051c4e81ea14146a9f1d82b7f5b5f1d P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New cstate: current_term: 1 leader_uuid: "0dc902d91ccc435a82a5ede4a7729811" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:57.303493 660 raft_consensus.cc:2932] T 5abcdc783c6144f28b9f8a9e1e162006 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.304864 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet 589fce6d925b43d895588bac355135b0 from peer 85f69eefe4294197923dff5efadcbbd5 ({username='slave'} at 127.22.242.72:54703)
I20240503 14:13:57.305038 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet 589fce6d925b43d895588bac355135b0 from peer 85f69eefe4294197923dff5efadcbbd5 at {username='slave'} at 127.22.242.72:54703: session id = 85f69eefe4294197923dff5efadcbbd5-589fce6d925b43d895588bac355135b0
I20240503 14:13:57.308342 31092 tablet_copy_source_session.cc:205] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:57.310061 31070 consensus_queue.cc:237] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:13:57.313294 30797 catalog_manager.cc:5484] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.316766 2445 raft_consensus.cc:1050] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER 4665a3377574417ba0d3165c236533c0 to VOTER
I20240503 14:13:57.321090 792 raft_consensus.cc:1261] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:57.324224 2143 raft_consensus.cc:2932] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:57.325851 30939 raft_consensus.cc:2932] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.327215 30940 raft_consensus.cc:1261] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.329306 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:57.287617 31326 raft_consensus.cc:1261] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:57.331689 2395 tablet_replica.cc:330] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be: stopping tablet replica
I20240503 14:13:57.331883 2395 raft_consensus.cc:2229] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:57.332130 2395 pending_rounds.cc:62] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be: Trying to abort 1 pending ops.
I20240503 14:13:57.332846 2138 consensus_queue.cc:1035] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:57.334229 30786 catalog_manager.cc:4830] TS 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063): tablet 91a7b8a53140429b956178d43478a541 (table rebalance_test_table_3 [id=ca6b9f7a0ded4c3aa1ea18aa2055c41b]) successfully deleted
I20240503 14:13:57.339027 2079 consensus_queue.cc:1035] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.340761 2445 consensus_queue.cc:237] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:13:57.343816 2446 consensus_queue.cc:1035] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:57.359381 2079 raft_consensus.cc:2932] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:57.365269 2426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 589fce6d925b43d895588bac355135b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:57.366374 792 raft_consensus.cc:2932] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:57.373500 2426 tablet_copy_client.cc:783] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:57.373838 2426 tablet_copy_client.cc:647] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:57.382350 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:57.398159 30794 catalog_manager.cc:5484] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 reported cstate change: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:57.402354 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:57.405140 31180 tablet_service.cc:1514] Processing DeleteTablet for tablet f3e80559017d4a6ba16d34e0b7aab085 with delete_type TABLET_DATA_TOMBSTONED (TS e7badbe63a8a4d9ca0c78818c6f4c8ce not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45854
I20240503 14:13:57.338634 31326 raft_consensus.cc:2932] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:57.381354 31331 consensus_queue.cc:237] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:13:57.381364 31328 raft_consensus.cc:1261] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.407627 2451 consensus_queue.cc:1035] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.381752 31329 raft_consensus.cc:2932] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:57.408813 31199 raft_consensus.cc:1261] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:57.409731 1722 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:57.410292 2395 pending_rounds.cc:68] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023598129201033216 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "f6aeaa3fe1cd4127a6144d85b0434057" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } } }
I20240503 14:13:57.410667 2395 raft_consensus.cc:2894] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 14:13:57.410770 2395 raft_consensus.cc:2866] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 14:13:57.410871 2395 raft_consensus.cc:2258] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:57.411470 31330 raft_consensus.cc:1261] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:57.411906 2395 ts_tablet_manager.cc:1875] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:57.412654 659 raft_consensus.cc:1261] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:57.413471 1722 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:57.416512 1722 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.420646 30786 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bc77a7da906b419085258f72667ff185 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:57.422322 659 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.424997 31199 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.428355 30798 catalog_manager.cc:5484] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0 reported cstate change: config changed from index 6 to 7, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) evicted. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.441828 30940 raft_consensus.cc:1261] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:57.443149 2451 consensus_queue.cc:1035] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:57.447638 659 raft_consensus.cc:1261] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.459556 1578 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:57.460306 2426 tablet_copy_client.cc:523] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:57.464874 2426 tablet_bootstrap.cc:492] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap starting.
I20240503 14:13:57.471061 2473 ts_tablet_manager.cc:902] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 4665a3377574417ba0d3165c236533c0: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:57.479465 31329 raft_consensus.cc:2932] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:57.485275 1628 raft_consensus.cc:1050] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce: attempting to promote NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 to VOTER
I20240503 14:13:57.486052 1628 consensus_queue.cc:237] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:13:57.476580 2451 consensus_queue.cc:1035] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20240503 14:13:57.494836 31199 raft_consensus.cc:1261] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.495735 30940 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.499533 1600 consensus_queue.cc:1035] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20240503 14:13:57.500453 31070 consensus_queue.cc:237] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:13:57.510303 2126 raft_consensus.cc:1050] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be: attempting to promote NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 to VOTER
I20240503 14:13:57.511137 2126 consensus_queue.cc:237] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:13:57.511857 31200 raft_consensus.cc:1261] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:57.511552 2451 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.512686 2446 consensus_queue.cc:1035] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:57.512836 2478 tablet_replica.cc:330] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce: stopping tablet replica
I20240503 14:13:57.513028 2478 raft_consensus.cc:2229] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:57.513300 2478 raft_consensus.cc:2258] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:57.515535 528 raft_consensus.cc:1261] T dd5869fd8f8d41d99cf531edb7df3750 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.515233 2446 raft_consensus.cc:2932] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.515895 2478 ts_tablet_manager.cc:1875] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:57.519383 30940 raft_consensus.cc:1261] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.519424 660 raft_consensus.cc:1261] T 5abcdc783c6144f28b9f8a9e1e162006 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:57.520067 1602 consensus_queue.cc:1035] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:57.521063 30787 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f3e80559017d4a6ba16d34e0b7aab085 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:57.522471 2090 consensus_queue.cc:1035] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20240503 14:13:57.523690 660 raft_consensus.cc:2932] T 5abcdc783c6144f28b9f8a9e1e162006 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.532989 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:57.533403 31200 raft_consensus.cc:2932] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.537432 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:57.537595 30787 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5abcdc783c6144f28b9f8a9e1e162006 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:57.539340 30940 raft_consensus.cc:1261] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.542951 30940 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.548173 527 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.549746 31072 raft_consensus.cc:1261] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.551249 2446 consensus_queue.cc:1035] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.560803 2069 raft_consensus.cc:2932] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.564472 31329 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.569312 2473 tablet_copy_client.cc:308] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 4665a3377574417ba0d3165c236533c0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:13:57.570420 30939 raft_consensus.cc:2932] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.575659 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet 4ff2fa69f79644c5b93beeba32fc9ca0 from peer 4665a3377574417ba0d3165c236533c0 ({username='slave'} at 127.22.242.71:41641)
I20240503 14:13:57.575855 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet 4ff2fa69f79644c5b93beeba32fc9ca0 from peer 4665a3377574417ba0d3165c236533c0 at {username='slave'} at 127.22.242.71:41641: session id = 4665a3377574417ba0d3165c236533c0-4ff2fa69f79644c5b93beeba32fc9ca0
I20240503 14:13:57.535542 1628 consensus_queue.cc:1035] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20240503 14:13:57.576418 2090 consensus_queue.cc:1035] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.578969 31092 tablet_copy_source_session.cc:205] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:57.578923 660 raft_consensus.cc:2932] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.586515 507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:57.592554 2478 ts_tablet_manager.cc:1888] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:57.592784 2478 log.cc:1196] T f3e80559017d4a6ba16d34e0b7aab085 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/f3e80559017d4a6ba16d34e0b7aab085
I20240503 14:13:57.599406 2473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff2fa69f79644c5b93beeba32fc9ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:57.603147 2294 raft_consensus.cc:2932] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.604975 31352 tablet_copy_service.cc:140] P 82cfdc130877418dbe9d980d368973be: Received BeginTabletCopySession request for tablet bc77a7da906b419085258f72667ff185 from peer 6967bdda50404d68a5ac043b709440e1 ({username='slave'} at 127.22.242.69:57313)
I20240503 14:13:57.605132 31352 tablet_copy_service.cc:161] P 82cfdc130877418dbe9d980d368973be: Beginning new tablet copy session on tablet bc77a7da906b419085258f72667ff185 from peer 6967bdda50404d68a5ac043b709440e1 at {username='slave'} at 127.22.242.69:57313: session id = 6967bdda50404d68a5ac043b709440e1-bc77a7da906b419085258f72667ff185
I20240503 14:13:57.605362 30798 catalog_manager.cc:5484] T 5abcdc783c6144f28b9f8a9e1e162006 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.608011 31352 tablet_copy_source_session.cc:205] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:57.609366 30920 tablet_service.cc:1514] Processing DeleteTablet for tablet 5abcdc783c6144f28b9f8a9e1e162006 with delete_type TABLET_DATA_TOMBSTONED (TS 0dc902d91ccc435a82a5ede4a7729811 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56642
I20240503 14:13:57.609210 527 raft_consensus.cc:2932] T dd5869fd8f8d41d99cf531edb7df3750 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.612583 31070 raft_consensus.cc:2932] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.613803 30798 catalog_manager.cc:5484] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:57.617069 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc77a7da906b419085258f72667ff185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:57.615619 30785 catalog_manager.cc:4830] TS e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297): tablet f3e80559017d4a6ba16d34e0b7aab085 (table rebalance_test_table_3 [id=ca6b9f7a0ded4c3aa1ea18aa2055c41b]) successfully deleted
I20240503 14:13:57.622773 30797 catalog_manager.cc:5484] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 reported cstate change: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.627557 2395 ts_tablet_manager.cc:1888] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:57.628008 2395 log.cc:1196] T f6aeaa3fe1cd4127a6144d85b0434057 P 82cfdc130877418dbe9d980d368973be: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/wals/f6aeaa3fe1cd4127a6144d85b0434057
I20240503 14:13:57.629527 30786 catalog_manager.cc:4830] TS 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063): tablet f6aeaa3fe1cd4127a6144d85b0434057 (table rebalance_test_table_2 [id=026356e4d1494d1c8649c4b1d401565d]) successfully deleted
I20240503 14:13:57.630661 2473 tablet_copy_client.cc:783] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:57.631668 30792 catalog_manager.cc:5484] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.632779 2473 tablet_copy_client.cc:647] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:57.628646 30940 raft_consensus.cc:2932] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.649580 31200 consensus_queue.cc:237] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:57.653314 30939 raft_consensus.cc:1261] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:57.653388 31138 consensus_peers.cc:485] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189): Couldn't send request to peer 6967bdda50404d68a5ac043b709440e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3330e28c2d7434699b78070dd92ed73. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:57.655895 2090 consensus_queue.cc:1035] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:13:57.657241 31070 raft_consensus.cc:1261] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:57.657927 1628 consensus_queue.cc:1035] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:57.664580 2509 ts_tablet_manager.cc:902] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 4665a3377574417ba0d3165c236533c0: Initiating tablet copy from peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:13:57.674049 1628 raft_consensus.cc:2932] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:57.674878 527 raft_consensus.cc:1261] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.675581 1578 consensus_queue.cc:1035] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.676363 30936 raft_consensus.cc:2932] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:57.678402 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:57.682870 31200 raft_consensus.cc:1261] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.683712 1578 consensus_queue.cc:1035] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.684600 31070 raft_consensus.cc:1261] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.684710 31072 raft_consensus.cc:2932] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:57.687065 2473 tablet_copy_client.cc:523] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 4665a3377574417ba0d3165c236533c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:57.689744 1578 consensus_queue.cc:1035] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.690961 2473 tablet_bootstrap.cc:492] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 4665a3377574417ba0d3165c236533c0: Bootstrap starting.
I20240503 14:13:57.691407 2406 tablet_copy_client.cc:783] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:57.691977 2406 tablet_copy_client.cc:647] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:57.721678 2509 tablet_copy_client.cc:308] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 4665a3377574417ba0d3165c236533c0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:46013
I20240503 14:13:57.736311 376 tablet_service.cc:1514] Processing DeleteTablet for tablet bc77a7da906b419085258f72667ff185 with delete_type TABLET_DATA_TOMBSTONED (TS 6967bdda50404d68a5ac043b709440e1 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55006
I20240503 14:13:57.736765 2406 tablet_copy_client.cc:523] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:57.738508 31202 raft_consensus.cc:2932] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.740777 2507 tablet_replica.cc:330] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811: stopping tablet replica
I20240503 14:13:57.740980 2507 raft_consensus.cc:2229] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:57.741189 2507 raft_consensus.cc:2258] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:57.742290 2507 ts_tablet_manager.cc:1875] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 14:13:57.754303 30787 catalog_manager.cc:4810] TS 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189): delete failed for tablet bc77a7da906b419085258f72667ff185 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc77a7da906b419085258f72667ff185 already in progress: copying tablet
I20240503 14:13:57.758082 31199 consensus_queue.cc:237] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:13:57.763792 1578 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.768846 31202 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.770406 527 raft_consensus.cc:1261] T dd5869fd8f8d41d99cf531edb7df3750 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:57.771086 1628 consensus_queue.cc:1035] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:57.774490 2294 raft_consensus.cc:2932] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.775084 528 raft_consensus.cc:2932] T dd5869fd8f8d41d99cf531edb7df3750 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.778645 30785 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dd5869fd8f8d41d99cf531edb7df3750 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:57.779773 526 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.789438 30792 catalog_manager.cc:5484] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index -1 to 4, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:57.790817 30794 catalog_manager.cc:5484] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:57.796594 31070 consensus_queue.cc:237] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:13:57.798228 31071 consensus_queue.cc:237] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:13:57.799093 30939 raft_consensus.cc:1261] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:57.799641 2090 consensus_queue.cc:1035] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:57.801064 30939 raft_consensus.cc:2932] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.801697 31329 raft_consensus.cc:1261] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:57.801857 30936 raft_consensus.cc:1261] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:57.802407 2446 consensus_queue.cc:1035] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:57.803589 2446 consensus_queue.cc:1035] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:57.804873 2446 raft_consensus.cc:2932] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.807194 2451 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.808933 30939 raft_consensus.cc:2932] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.809168 31329 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.812549 527 raft_consensus.cc:1261] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:57.813815 2446 consensus_queue.cc:1035] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:57.815460 527 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.818382 31071 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.827076 30787 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ebd6dc710e144e61a5fae3911f25ee5c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:57.827404 30787 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 584769402a7e4e8a985a8f2363e2b8b9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:57.845110 31329 consensus_queue.cc:237] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:13:57.845336 30794 catalog_manager.cc:5484] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.847600 660 raft_consensus.cc:1261] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:57.848248 2451 consensus_queue.cc:1035] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:57.848448 527 raft_consensus.cc:1261] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:57.850623 30797 catalog_manager.cc:5484] T dd5869fd8f8d41d99cf531edb7df3750 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.850831 660 raft_consensus.cc:2932] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.851200 30797 catalog_manager.cc:5484] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 5 to 6, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:57.851832 1578 consensus_queue.cc:1035] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:57.856087 1850 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.862190 527 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.873112 31200 raft_consensus.cc:1261] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:57.874109 2126 consensus_queue.cc:1035] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:57.877538 30786 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 23476c4bcf6e4b4085d59d29739b6edb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:57.879323 31180 tablet_service.cc:1514] Processing DeleteTablet for tablet ebd6dc710e144e61a5fae3911f25ee5c with delete_type TABLET_DATA_TOMBSTONED (TS e7badbe63a8a4d9ca0c78818c6f4c8ce not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45854
I20240503 14:13:57.879726 2478 tablet_replica.cc:330] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce: stopping tablet replica
I20240503 14:13:57.879930 2478 raft_consensus.cc:2229] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:57.880170 2478 raft_consensus.cc:2258] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:57.881264 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet dd5869fd8f8d41d99cf531edb7df3750 with delete_type TABLET_DATA_TOMBSTONED (TS 6dc4948ed7c740ab89b9e349d151b0e5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37226
I20240503 14:13:57.882155 2478 ts_tablet_manager.cc:1875] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:57.875818 31200 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.884142 2546 tablet_replica.cc:330] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5: stopping tablet replica
I20240503 14:13:57.884382 2546 raft_consensus.cc:2229] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:57.884636 2546 raft_consensus.cc:2258] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:57.887315 2546 ts_tablet_manager.cc:1875] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 14:13:57.887898 30787 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet ebd6dc710e144e61a5fae3911f25ee5c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 5 but the committed config has opid_index of 6
I20240503 14:13:57.890060 2478 ts_tablet_manager.cc:1888] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:57.890604 2478 log.cc:1196] T ebd6dc710e144e61a5fae3911f25ee5c P e7badbe63a8a4d9ca0c78818c6f4c8ce: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/ebd6dc710e144e61a5fae3911f25ee5c
I20240503 14:13:57.895880 30785 catalog_manager.cc:4830] TS e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297): tablet ebd6dc710e144e61a5fae3911f25ee5c (table rebalance_test_table_1 [id=f6d48b36155b41d3a75fe51d7d204ea7]) successfully deleted
I20240503 14:13:57.940637 30961 tablet_copy_service.cc:140] P 0dc902d91ccc435a82a5ede4a7729811: Received BeginTabletCopySession request for tablet 5051c4e81ea14146a9f1d82b7f5b5f1d from peer 4665a3377574417ba0d3165c236533c0 ({username='slave'} at 127.22.242.71:53061)
I20240503 14:13:57.940805 30961 tablet_copy_service.cc:161] P 0dc902d91ccc435a82a5ede4a7729811: Beginning new tablet copy session on tablet 5051c4e81ea14146a9f1d82b7f5b5f1d from peer 4665a3377574417ba0d3165c236533c0 at {username='slave'} at 127.22.242.71:53061: session id = 4665a3377574417ba0d3165c236533c0-5051c4e81ea14146a9f1d82b7f5b5f1d
I20240503 14:13:57.955752 2406 tablet_bootstrap.cc:492] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: Bootstrap starting.
I20240503 14:13:57.971776 30920 tablet_service.cc:1514] Processing DeleteTablet for tablet 584769402a7e4e8a985a8f2363e2b8b9 with delete_type TABLET_DATA_TOMBSTONED (TS 0dc902d91ccc435a82a5ede4a7729811 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56642
W20240503 14:13:57.973271 30786 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 23476c4bcf6e4b4085d59d29739b6edb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 5 but the committed config has opid_index of 6
I20240503 14:13:57.974830 30793 catalog_manager.cc:5484] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 5 to 6, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:13:57.983830 30961 tablet_copy_source_session.cc:205] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:57.985587 2509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5051c4e81ea14146a9f1d82b7f5b5f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:57.997418 2509 tablet_copy_client.cc:783] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:57.997740 2509 tablet_copy_client.cc:647] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:57.999977 2509 tablet_copy_client.cc:523] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 4665a3377574417ba0d3165c236533c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:58.001446 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet 23476c4bcf6e4b4085d59d29739b6edb with delete_type TABLET_DATA_TOMBSTONED (TS 6dc4948ed7c740ab89b9e349d151b0e5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37226
I20240503 14:13:58.003278 2507 ts_tablet_manager.cc:1888] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:58.005057 2507 log.cc:1196] T 5abcdc783c6144f28b9f8a9e1e162006 P 0dc902d91ccc435a82a5ede4a7729811: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/5abcdc783c6144f28b9f8a9e1e162006
I20240503 14:13:58.005679 2546 ts_tablet_manager.cc:1888] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:58.006011 30784 catalog_manager.cc:4830] TS 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): tablet 5abcdc783c6144f28b9f8a9e1e162006 (table rebalance_test_table_0 [id=fa8e781fa3e140b6865ce5f7b429e4c4]) successfully deleted
I20240503 14:13:58.006551 2507 tablet_replica.cc:330] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811: stopping tablet replica
I20240503 14:13:58.006834 2507 raft_consensus.cc:2229] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:58.008419 2507 raft_consensus.cc:2258] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:58.009539 2507 ts_tablet_manager.cc:1875] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:58.013686 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:58.023543 2509 tablet_bootstrap.cc:492] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 4665a3377574417ba0d3165c236533c0: Bootstrap starting.
I20240503 14:13:58.047237 2546 log.cc:1196] T dd5869fd8f8d41d99cf531edb7df3750 P 6dc4948ed7c740ab89b9e349d151b0e5: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/dd5869fd8f8d41d99cf531edb7df3750
I20240503 14:13:58.047119 31330 consensus_queue.cc:237] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:13:58.048323 30787 catalog_manager.cc:4830] TS 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221): tablet dd5869fd8f8d41d99cf531edb7df3750 (table rebalance_test_table_4 [id=f05910cae90c4382a503986593cff68f]) successfully deleted
I20240503 14:13:58.050655 660 raft_consensus.cc:1261] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:58.051754 30940 raft_consensus.cc:1261] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:58.051643 2126 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:58.052382 2126 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:58.055635 2126 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:58.056651 2546 tablet_replica.cc:330] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5: stopping tablet replica
I20240503 14:13:58.056860 30940 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:58.057514 2546 raft_consensus.cc:2229] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:58.057857 2546 raft_consensus.cc:2258] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:58.059123 660 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:58.059336 2546 ts_tablet_manager.cc:1875] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:58.073549 2507 ts_tablet_manager.cc:1888] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:58.074750 30786 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bc77a7da906b419085258f72667ff185 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:58.089552 30796 catalog_manager.cc:5484] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index 7 to 8, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:58.098031 31180 tablet_service.cc:1514] Processing DeleteTablet for tablet bc77a7da906b419085258f72667ff185 with delete_type TABLET_DATA_TOMBSTONED (TS e7badbe63a8a4d9ca0c78818c6f4c8ce not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:45854
I20240503 14:13:58.098538 2478 tablet_replica.cc:330] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce: stopping tablet replica
I20240503 14:13:58.098757 2478 raft_consensus.cc:2229] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:58.099084 2478 raft_consensus.cc:2258] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:58.100281 2478 ts_tablet_manager.cc:1875] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:58.103418 2546 ts_tablet_manager.cc:1888] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:58.103610 2546 log.cc:1196] T 23476c4bcf6e4b4085d59d29739b6edb P 6dc4948ed7c740ab89b9e349d151b0e5: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/23476c4bcf6e4b4085d59d29739b6edb
I20240503 14:13:58.104426 30787 catalog_manager.cc:4830] TS 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221): tablet 23476c4bcf6e4b4085d59d29739b6edb (table rebalance_test_table_4 [id=f05910cae90c4382a503986593cff68f]) successfully deleted
I20240503 14:13:58.106899 2478 ts_tablet_manager.cc:1888] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 14:13:58.107097 2478 log.cc:1196] T bc77a7da906b419085258f72667ff185 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/bc77a7da906b419085258f72667ff185
I20240503 14:13:58.108208 30785 catalog_manager.cc:4830] TS e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297): tablet bc77a7da906b419085258f72667ff185 (table rebalance_test_table_5 [id=04f8dfb749c04486895d6b0f7c206446]) successfully deleted
I20240503 14:13:58.118986 2426 tablet_bootstrap.cc:492] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:58.119438 2426 tablet_bootstrap.cc:492] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap complete.
I20240503 14:13:58.119722 2426 ts_tablet_manager.cc:1367] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5: Time spent bootstrapping tablet: real 0.655s user 0.062s sys 0.056s
I20240503 14:13:58.120757 2426 raft_consensus.cc:348] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:58.121044 2426 raft_consensus.cc:729] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85f69eefe4294197923dff5efadcbbd5, State: Initialized, Role: LEARNER
I20240503 14:13:58.121347 2426 consensus_queue.cc:260] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:58.134047 2507 log.cc:1196] T 584769402a7e4e8a985a8f2363e2b8b9 P 0dc902d91ccc435a82a5ede4a7729811: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/584769402a7e4e8a985a8f2363e2b8b9
I20240503 14:13:58.135327 30784 catalog_manager.cc:4830] TS 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): tablet 584769402a7e4e8a985a8f2363e2b8b9 (table rebalance_test_table_8 [id=c243ced42cda4e3996901964941b50f4]) successfully deleted
I20240503 14:13:58.139688 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:58.146653 2426 ts_tablet_manager.cc:1398] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5: Time spent starting tablet: real 0.027s user 0.003s sys 0.006s
I20240503 14:13:58.147446 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session 85f69eefe4294197923dff5efadcbbd5-589fce6d925b43d895588bac355135b0 received from {username='slave'} at 127.22.242.72:54703
I20240503 14:13:58.147584 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session 85f69eefe4294197923dff5efadcbbd5-589fce6d925b43d895588bac355135b0 on tablet 589fce6d925b43d895588bac355135b0 with peer 85f69eefe4294197923dff5efadcbbd5
I20240503 14:13:58.177577 2473 tablet_bootstrap.cc:492] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 4665a3377574417ba0d3165c236533c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:58.177807 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:58.178085 2473 tablet_bootstrap.cc:492] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 4665a3377574417ba0d3165c236533c0: Bootstrap complete.
I20240503 14:13:58.189316 2473 ts_tablet_manager.cc:1367] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 4665a3377574417ba0d3165c236533c0: Time spent bootstrapping tablet: real 0.498s user 0.052s sys 0.006s
I20240503 14:13:58.192214 2473 raft_consensus.cc:348] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:58.192517 2473 raft_consensus.cc:729] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4665a3377574417ba0d3165c236533c0, State: Initialized, Role: LEARNER
I20240503 14:13:58.192723 2473 consensus_queue.cc:260] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 4665a3377574417ba0d3165c236533c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:58.193316 2473 ts_tablet_manager.cc:1398] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 4665a3377574417ba0d3165c236533c0: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 14:13:58.220315 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session 4665a3377574417ba0d3165c236533c0-4ff2fa69f79644c5b93beeba32fc9ca0 received from {username='slave'} at 127.22.242.71:41641
I20240503 14:13:58.220454 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session 4665a3377574417ba0d3165c236533c0-4ff2fa69f79644c5b93beeba32fc9ca0 on tablet 4ff2fa69f79644c5b93beeba32fc9ca0 with peer 4665a3377574417ba0d3165c236533c0
I20240503 14:13:58.253687 2576 ts_tablet_manager.cc:902] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1: Initiating tablet copy from peer e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:58.279574 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:58.364331 2576 tablet_copy_client.cc:308] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44297
I20240503 14:13:58.365473 31221 tablet_copy_service.cc:140] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Received BeginTabletCopySession request for tablet f3330e28c2d7434699b78070dd92ed73 from peer 6967bdda50404d68a5ac043b709440e1 ({username='slave'} at 127.22.242.69:38549)
I20240503 14:13:58.365646 31221 tablet_copy_service.cc:161] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Beginning new tablet copy session on tablet f3330e28c2d7434699b78070dd92ed73 from peer 6967bdda50404d68a5ac043b709440e1 at {username='slave'} at 127.22.242.69:38549: session id = 6967bdda50404d68a5ac043b709440e1-f3330e28c2d7434699b78070dd92ed73
I20240503 14:13:58.368881 31221 tablet_copy_source_session.cc:205] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:58.370446 2576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3330e28c2d7434699b78070dd92ed73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:58.381992 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:58.390305 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:58.398005 2509 tablet_bootstrap.cc:492] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 4665a3377574417ba0d3165c236533c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:58.398628 2509 tablet_bootstrap.cc:492] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 4665a3377574417ba0d3165c236533c0: Bootstrap complete.
I20240503 14:13:58.398975 2509 ts_tablet_manager.cc:1367] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 4665a3377574417ba0d3165c236533c0: Time spent bootstrapping tablet: real 0.376s user 0.054s sys 0.002s
I20240503 14:13:58.399822 2509 raft_consensus.cc:348] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:58.400210 2509 raft_consensus.cc:729] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4665a3377574417ba0d3165c236533c0, State: Initialized, Role: LEARNER
I20240503 14:13:58.400537 2509 consensus_queue.cc:260] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 4665a3377574417ba0d3165c236533c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:58.401181 660 raft_consensus.cc:1203] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:58.401336 2509 ts_tablet_manager.cc:1398] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 4665a3377574417ba0d3165c236533c0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:58.402254 30961 tablet_copy_service.cc:342] P 0dc902d91ccc435a82a5ede4a7729811: Request end of tablet copy session 4665a3377574417ba0d3165c236533c0-5051c4e81ea14146a9f1d82b7f5b5f1d received from {username='slave'} at 127.22.242.71:53061
I20240503 14:13:58.402416 30961 tablet_copy_service.cc:433] P 0dc902d91ccc435a82a5ede4a7729811: ending tablet copy session 4665a3377574417ba0d3165c236533c0-5051c4e81ea14146a9f1d82b7f5b5f1d on tablet 5051c4e81ea14146a9f1d82b7f5b5f1d with peer 4665a3377574417ba0d3165c236533c0
I20240503 14:13:58.403568 2576 tablet_copy_client.cc:783] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:58.404946 2576 tablet_copy_client.cc:647] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:58.410346 2576 tablet_copy_client.cc:523] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:58.414106 2576 tablet_bootstrap.cc:492] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1: Bootstrap starting.
I20240503 14:13:58.430801 2406 tablet_bootstrap.cc:492] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:58.434134 507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:58.472632 2406 tablet_bootstrap.cc:492] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: Bootstrap complete.
I20240503 14:13:58.473054 2406 ts_tablet_manager.cc:1367] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: Time spent bootstrapping tablet: real 0.517s user 0.052s sys 0.010s
I20240503 14:13:58.474128 2406 raft_consensus.cc:348] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:13:58.474463 2406 raft_consensus.cc:729] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: 6967bdda50404d68a5ac043b709440e1, State: Initialized, Role: NON_PARTICIPANT
I20240503 14:13:58.474788 2406 consensus_queue.cc:260] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:13:58.478874 2576 tablet_bootstrap.cc:492] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:58.479422 2576 tablet_bootstrap.cc:492] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1: Bootstrap complete.
I20240503 14:13:58.479876 2576 ts_tablet_manager.cc:1367] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1: Time spent bootstrapping tablet: real 0.066s user 0.054s sys 0.009s
I20240503 14:13:58.481267 2576 raft_consensus.cc:348] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:58.481604 2576 raft_consensus.cc:729] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6967bdda50404d68a5ac043b709440e1, State: Initialized, Role: LEARNER
I20240503 14:13:58.481863 2576 consensus_queue.cc:260] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:58.489238 2576 ts_tablet_manager.cc:1398] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1: Time spent starting tablet: real 0.009s user 0.001s sys 0.007s
I20240503 14:13:58.489989 31221 tablet_copy_service.cc:342] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Request end of tablet copy session 6967bdda50404d68a5ac043b709440e1-f3330e28c2d7434699b78070dd92ed73 received from {username='slave'} at 127.22.242.69:38549
I20240503 14:13:58.490149 31221 tablet_copy_service.cc:433] P e7badbe63a8a4d9ca0c78818c6f4c8ce: ending tablet copy session 6967bdda50404d68a5ac043b709440e1-f3330e28c2d7434699b78070dd92ed73 on tablet f3330e28c2d7434699b78070dd92ed73 with peer 6967bdda50404d68a5ac043b709440e1
I20240503 14:13:58.492401 376 tablet_service.cc:1514] Processing DeleteTablet for tablet bc77a7da906b419085258f72667ff185 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 7 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:55006
W20240503 14:13:58.494961 30787 catalog_manager.cc:4810] TS 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189): delete failed for tablet bc77a7da906b419085258f72667ff185 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc77a7da906b419085258f72667ff185 already in progress: copying tablet
I20240503 14:13:58.497339 2406 ts_tablet_manager.cc:1398] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: Time spent starting tablet: real 0.024s user 0.008s sys 0.001s
I20240503 14:13:58.498880 31352 tablet_copy_service.cc:342] P 82cfdc130877418dbe9d980d368973be: Request end of tablet copy session 6967bdda50404d68a5ac043b709440e1-bc77a7da906b419085258f72667ff185 received from {username='slave'} at 127.22.242.69:57313
I20240503 14:13:58.499037 31352 tablet_copy_service.cc:433] P 82cfdc130877418dbe9d980d368973be: ending tablet copy session 6967bdda50404d68a5ac043b709440e1-bc77a7da906b419085258f72667ff185 on tablet bc77a7da906b419085258f72667ff185 with peer 6967bdda50404d68a5ac043b709440e1
I20240503 14:13:58.500485 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:58.503973 376 tablet_service.cc:1514] Processing DeleteTablet for tablet bc77a7da906b419085258f72667ff185 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 7 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:55006
I20240503 14:13:58.513761 792 raft_consensus.cc:1203] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:58.516917 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:58.519577 2608 tablet_replica.cc:330] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: stopping tablet replica
I20240503 14:13:58.520314 2608 raft_consensus.cc:2229] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20240503 14:13:58.522006 2608 raft_consensus.cc:2258] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20240503 14:13:58.527192 2608 ts_tablet_manager.cc:1875] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:58.539475 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:58.563112 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:58.576277 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:58.583895 2608 ts_tablet_manager.cc:1888] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 14:13:58.584100 2608 log.cc:1196] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal/wals/bc77a7da906b419085258f72667ff185
I20240503 14:13:58.596482 30787 catalog_manager.cc:4830] TS 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189): tablet bc77a7da906b419085258f72667ff185 (table rebalance_test_table_5 [id=04f8dfb749c04486895d6b0f7c206446]) successfully deleted
I20240503 14:13:58.598518 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:58.604454 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:58.623607 507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:58.631367 31072 consensus_queue.cc:237] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
W20240503 14:13:58.664481 31008 consensus_peers.cc:485] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189): Couldn't send request to peer 6967bdda50404d68a5ac043b709440e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c837def6d76242f08b20a448541e64d6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:58.664750 31200 raft_consensus.cc:1261] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:58.666260 527 raft_consensus.cc:1261] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:58.665416 2451 consensus_queue.cc:1035] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240503 14:13:58.669405 2451 consensus_queue.cc:1035] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240503 14:13:58.677310 2451 raft_consensus.cc:2932] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:58.680583 31200 raft_consensus.cc:2932] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:58.693286 30791 catalog_manager.cc:5484] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 6 to 7, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:58.698314 660 raft_consensus.cc:1203] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:58.699719 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:58.731508 527 raft_consensus.cc:2932] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:13:58.747781 393 raft_consensus.cc:1203] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:58.806860 31200 consensus_queue.cc:237] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:13:58.822587 393 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:13:58.823508 2637 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 14:13:58.835986 31135 consensus_peers.cc:485] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829): Couldn't send request to peer fa71cbdf0e544b81bd4e7e27196d2c91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f82073122a74268a464387659dd90bf. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:58.837961 31330 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:13:58.838760 2637 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:58.870579 2637 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:58.871863 394 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:58.883950 30792 catalog_manager.cc:5484] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 8 to 9, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:58.897792 31329 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:13:58.917919 2446 raft_consensus.cc:1050] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER 4665a3377574417ba0d3165c236533c0 to VOTER
I20240503 14:13:58.918759 2446 consensus_queue.cc:237] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:13:58.925869 2451 raft_consensus.cc:1050] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER 85f69eefe4294197923dff5efadcbbd5 to VOTER
I20240503 14:13:58.926678 2451 consensus_queue.cc:237] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:13:58.937217 660 raft_consensus.cc:1261] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:58.938058 2079 consensus_queue.cc:1035] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:58.939956 792 raft_consensus.cc:1261] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:58.941838 30940 raft_consensus.cc:1261] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:58.941857 2079 consensus_queue.cc:1035] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:58.942592 2079 consensus_queue.cc:1035] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:58.941254 31330 raft_consensus.cc:1261] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:58.944615 2079 consensus_queue.cc:1035] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:58.964826 30938 raft_consensus.cc:1261] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:58.966267 2671 consensus_queue.cc:1035] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:58.970630 31329 raft_consensus.cc:1261] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:58.972152 2451 consensus_queue.cc:1035] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:58.987258 2446 raft_consensus.cc:2932] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:58.992060 30936 raft_consensus.cc:2932] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:58.994911 31329 raft_consensus.cc:2932] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:59.001581 30800 catalog_manager.cc:5484] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:59.007800 31329 consensus_queue.cc:237] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:59.036746 2671 raft_consensus.cc:2932] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:59.043022 31072 consensus_queue.cc:237] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:13:59.045145 659 raft_consensus.cc:2932] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
W20240503 14:13:59.050290 31268 consensus_peers.cc:485] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55cf34de74694f47ad3f9f4d9bdffae4. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:59.050383 31330 raft_consensus.cc:1261] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:59.051751 2446 consensus_queue.cc:1035] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:59.054349 2446 raft_consensus.cc:2932] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:59.053890 31330 raft_consensus.cc:2932] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:59.057835 31326 raft_consensus.cc:2932] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:59.060786 30791 catalog_manager.cc:5484] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:59.062057 30787 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4ff2fa69f79644c5b93beeba32fc9ca0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:59.064808 658 raft_consensus.cc:1261] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:59.066279 2451 consensus_queue.cc:1035] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:59.067734 30800 catalog_manager.cc:5484] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:59.067833 658 raft_consensus.cc:2932] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:59.071935 792 raft_consensus.cc:2932] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:59.073331 30920 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ff2fa69f79644c5b93beeba32fc9ca0 with delete_type TABLET_DATA_TOMBSTONED (TS 0dc902d91ccc435a82a5ede4a7729811 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56642
I20240503 14:13:59.076834 31200 consensus_queue.cc:237] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:13:59.082805 660 raft_consensus.cc:1261] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:59.083408 1722 consensus_queue.cc:1035] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:59.085801 31072 consensus_queue.cc:237] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:13:59.088991 30936 raft_consensus.cc:1261] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:59.090041 2079 consensus_queue.cc:1035] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:59.091377 30936 raft_consensus.cc:2932] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:59.092272 30938 raft_consensus.cc:1261] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:59.092994 1722 consensus_queue.cc:1035] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:59.094471 2143 raft_consensus.cc:1050] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811: attempting to promote NON_VOTER 4665a3377574417ba0d3165c236533c0 to VOTER
I20240503 14:13:59.095245 2143 consensus_queue.cc:237] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:13:59.098678 660 raft_consensus.cc:1261] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:59.098932 31199 raft_consensus.cc:1261] T 5051c4e81ea14146a9f1d82b7f5b5f1d P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:59.099169 2699 consensus_queue.cc:1035] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:59.099900 2633 consensus_queue.cc:1035] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:59.099680 1722 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:59.103385 30938 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:59.112066 31326 raft_consensus.cc:1261] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:59.113042 2143 consensus_queue.cc:1035] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:59.113157 2451 raft_consensus.cc:2932] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:59.116395 30793 catalog_manager.cc:5484] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:59.118793 30787 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 589fce6d925b43d895588bac355135b0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:59.120899 30938 raft_consensus.cc:2932] T 589fce6d925b43d895588bac355135b0 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:59.122201 30798 catalog_manager.cc:5484] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:59.126796 31311 tablet_service.cc:1514] Processing DeleteTablet for tablet 589fce6d925b43d895588bac355135b0 with delete_type TABLET_DATA_TOMBSTONED (TS 82cfdc130877418dbe9d980d368973be not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48102
I20240503 14:13:59.167454 31072 consensus_queue.cc:237] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
W20240503 14:13:59.176960 31138 consensus_peers.cc:485] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43221a82a1124bf8840c6369f27f47f2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:59.177824 2692 ts_tablet_manager.cc:902] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:59.179801 527 raft_consensus.cc:1261] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:59.180716 2637 consensus_queue.cc:1035] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:59.184239 658 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:59.186658 659 raft_consensus.cc:1261] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 14:13:59.187184 31008 consensus_peers.cc:485] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebd6dc710e144e61a5fae3911f25ee5c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:59.187353 2446 consensus_queue.cc:1035] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:59.188356 790 raft_consensus.cc:1261] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:59.189766 2446 consensus_queue.cc:1035] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:59.189600 2143 raft_consensus.cc:2932] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:59.193742 2637 raft_consensus.cc:2932] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:59.195089 792 raft_consensus.cc:2932] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:13:59.196884 528 raft_consensus.cc:2932] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:59.198340 2691 tablet_replica.cc:330] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811: stopping tablet replica
I20240503 14:13:59.200755 2691 raft_consensus.cc:2229] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:59.198515 30940 raft_consensus.cc:1261] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:59.202314 2691 raft_consensus.cc:2258] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:59.202493 2446 consensus_queue.cc:1035] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:59.204100 2691 ts_tablet_manager.cc:1875] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:59.207422 30800 catalog_manager.cc:5484] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91 reported cstate change: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:59.209827 31200 raft_consensus.cc:2932] T 5051c4e81ea14146a9f1d82b7f5b5f1d P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:59.199972 30791 catalog_manager.cc:5484] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0dc902d91ccc435a82a5ede4a7729811" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:59.214767 660 raft_consensus.cc:2932] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:59.189595 30938 raft_consensus.cc:1261] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:59.224884 2637 consensus_queue.cc:1035] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:59.228749 2451 raft_consensus.cc:2932] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:59.229451 30936 raft_consensus.cc:2932] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:59.236191 30792 catalog_manager.cc:5484] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:59.237097 30938 raft_consensus.cc:2932] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:59.251782 31072 consensus_queue.cc:237] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:59.252511 2692 tablet_copy_client.cc:308] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:13:59.253808 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet c837def6d76242f08b20a448541e64d6 from peer 6967bdda50404d68a5ac043b709440e1 ({username='slave'} at 127.22.242.69:44021)
I20240503 14:13:59.253960 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet c837def6d76242f08b20a448541e64d6 from peer 6967bdda50404d68a5ac043b709440e1 at {username='slave'} at 127.22.242.69:44021: session id = 6967bdda50404d68a5ac043b709440e1-c837def6d76242f08b20a448541e64d6
I20240503 14:13:59.257395 31092 tablet_copy_source_session.cc:205] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:59.258805 2692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c837def6d76242f08b20a448541e64d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:59.269170 2692 tablet_copy_client.cc:783] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:59.269685 2692 tablet_copy_client.cc:647] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:59.274109 2692 tablet_copy_client.cc:523] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:59.274019 30938 consensus_queue.cc:237] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:13:59.276403 31200 raft_consensus.cc:1261] T 5051c4e81ea14146a9f1d82b7f5b5f1d P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:59.277671 660 raft_consensus.cc:1261] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:59.278208 2633 consensus_queue.cc:1035] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:59.278447 2692 tablet_bootstrap.cc:492] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1: Bootstrap starting.
I20240503 14:13:59.278851 2143 consensus_queue.cc:1035] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:13:59.282979 31008 consensus_peers.cc:485] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da9dd5cd81114817b0f898c6b8fb3c07. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:59.284879 31330 raft_consensus.cc:1261] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:59.285068 2633 raft_consensus.cc:2932] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:59.285409 2671 consensus_queue.cc:1035] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:59.289239 31200 raft_consensus.cc:2932] T 5051c4e81ea14146a9f1d82b7f5b5f1d P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:59.290632 660 raft_consensus.cc:2932] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:13:59.292716 30938 raft_consensus.cc:1261] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:59.293290 2671 consensus_queue.cc:1035] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:59.295019 2691 ts_tablet_manager.cc:1888] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:59.298667 2705 tablet_replica.cc:330] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be: stopping tablet replica
I20240503 14:13:59.298931 2705 raft_consensus.cc:2229] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:59.299407 2705 raft_consensus.cc:2258] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:59.300381 30784 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5051c4e81ea14146a9f1d82b7f5b5f1d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:59.302919 2691 log.cc:1196] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 0dc902d91ccc435a82a5ede4a7729811: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/4ff2fa69f79644c5b93beeba32fc9ca0
I20240503 14:13:59.302987 30798 catalog_manager.cc:5484] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 5 to 6, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New cstate: current_term: 1 leader_uuid: "0dc902d91ccc435a82a5ede4a7729811" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:59.304659 30784 catalog_manager.cc:4830] TS 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): tablet 4ff2fa69f79644c5b93beeba32fc9ca0 (table rebalance_test_table_9 [id=acc4ce6b628f4f498f830c4572e30d65]) successfully deleted
I20240503 14:13:59.316427 2705 ts_tablet_manager.cc:1875] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:59.319907 31311 tablet_service.cc:1514] Processing DeleteTablet for tablet 5051c4e81ea14146a9f1d82b7f5b5f1d with delete_type TABLET_DATA_TOMBSTONED (TS 82cfdc130877418dbe9d980d368973be not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48102
I20240503 14:13:59.320888 2671 raft_consensus.cc:2932] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:59.324940 30936 raft_consensus.cc:2932] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:59.327659 659 raft_consensus.cc:2932] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:13:59.334934 30798 catalog_manager.cc:5484] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:59.345302 2090 raft_consensus.cc:1050] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce: attempting to promote NON_VOTER 6967bdda50404d68a5ac043b709440e1 to VOTER
I20240503 14:13:59.345396 2705 ts_tablet_manager.cc:1888] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:59.345513 2705 log.cc:1196] T 589fce6d925b43d895588bac355135b0 P 82cfdc130877418dbe9d980d368973be: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/wals/589fce6d925b43d895588bac355135b0
I20240503 14:13:59.346139 2090 consensus_queue.cc:237] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:13:59.347338 30786 catalog_manager.cc:4830] TS 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063): tablet 589fce6d925b43d895588bac355135b0 (table rebalance_test_table_6 [id=08c2ccf24fcc40f9a0fd947d52b50f8e]) successfully deleted
I20240503 14:13:59.347510 2705 tablet_replica.cc:330] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be: stopping tablet replica
I20240503 14:13:59.348091 2705 raft_consensus.cc:2229] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:59.348824 2705 pending_rounds.cc:62] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be: Trying to abort 1 pending ops.
I20240503 14:13:59.349049 2705 pending_rounds.cc:68] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023598137732071424 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "5051c4e81ea14146a9f1d82b7f5b5f1d" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } } }
I20240503 14:13:59.349952 2705 raft_consensus.cc:2894] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 14:13:59.350297 2705 raft_consensus.cc:2866] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 14:13:59.350605 2705 raft_consensus.cc:2258] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:59.351819 2705 ts_tablet_manager.cc:1875] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:59.356009 2692 tablet_bootstrap.cc:492] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:59.356489 2692 tablet_bootstrap.cc:492] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1: Bootstrap complete.
I20240503 14:13:59.356777 2692 ts_tablet_manager.cc:1367] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1: Time spent bootstrapping tablet: real 0.078s user 0.059s sys 0.000s
I20240503 14:13:59.357806 31330 raft_consensus.cc:2932] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:59.357584 2692 raft_consensus.cc:348] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:59.358726 2692 raft_consensus.cc:729] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6967bdda50404d68a5ac043b709440e1, State: Initialized, Role: LEARNER
I20240503 14:13:59.358958 2692 consensus_queue.cc:260] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:59.360394 2692 ts_tablet_manager.cc:1398] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:59.361176 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session 6967bdda50404d68a5ac043b709440e1-c837def6d76242f08b20a448541e64d6 received from {username='slave'} at 127.22.242.69:44021
I20240503 14:13:59.361313 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session 6967bdda50404d68a5ac043b709440e1-c837def6d76242f08b20a448541e64d6 on tablet c837def6d76242f08b20a448541e64d6 with peer 6967bdda50404d68a5ac043b709440e1
I20240503 14:13:59.371821 393 raft_consensus.cc:1261] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:59.372020 2705 ts_tablet_manager.cc:1888] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:59.372195 2705 log.cc:1196] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 82cfdc130877418dbe9d980d368973be: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/wals/5051c4e81ea14146a9f1d82b7f5b5f1d
I20240503 14:13:59.372442 2639 consensus_queue.cc:1035] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:59.373093 30786 catalog_manager.cc:4830] TS 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063): tablet 5051c4e81ea14146a9f1d82b7f5b5f1d (table rebalance_test_table_2 [id=026356e4d1494d1c8649c4b1d401565d]) successfully deleted
I20240503 14:13:59.379630 31072 raft_consensus.cc:1261] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:59.380859 2637 consensus_queue.cc:1035] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:59.389997 31330 consensus_queue.cc:237] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:13:59.392195 792 raft_consensus.cc:1261] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:13:59.392524 31072 raft_consensus.cc:1261] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:13:59.392989 2706 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:59.393990 30938 raft_consensus.cc:1261] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:59.394579 2639 consensus_queue.cc:1035] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:59.398360 2685 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:59.404623 2637 raft_consensus.cc:2932] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:59.408913 394 raft_consensus.cc:2932] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:59.410838 2707 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:59.411710 31071 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:59.417006 30792 catalog_manager.cc:5484] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index 8 to 9, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:59.419178 30798 catalog_manager.cc:5484] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1 reported cstate change: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:59.422437 790 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:13:59.437041 2750 ts_tablet_manager.cc:902] T 2f82073122a74268a464387659dd90bf P fa71cbdf0e544b81bd4e7e27196d2c91: Initiating tablet copy from peer e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
W20240503 14:13:59.453969 31268 consensus_peers.cc:485] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 626fcb185af0446b89ae82066e08981d. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:59.463814 31138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.69:42189, user_credentials={real_user=slave}} blocked reactor thread for 46486.7us
I20240503 14:13:59.464380 30936 raft_consensus.cc:2932] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:59.464728 31072 raft_consensus.cc:2932] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:13:59.563063 393 raft_consensus.cc:1203] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:13:59.572430 2750 tablet_copy_client.cc:308] T 2f82073122a74268a464387659dd90bf P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44297
I20240503 14:13:59.573431 31221 tablet_copy_service.cc:140] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Received BeginTabletCopySession request for tablet 2f82073122a74268a464387659dd90bf from peer fa71cbdf0e544b81bd4e7e27196d2c91 ({username='slave'} at 127.22.242.70:33209)
I20240503 14:13:59.573664 31221 tablet_copy_service.cc:161] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Beginning new tablet copy session on tablet 2f82073122a74268a464387659dd90bf from peer fa71cbdf0e544b81bd4e7e27196d2c91 at {username='slave'} at 127.22.242.70:33209: session id = fa71cbdf0e544b81bd4e7e27196d2c91-2f82073122a74268a464387659dd90bf
I20240503 14:13:59.578725 31221 tablet_copy_source_session.cc:205] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:59.583204 2750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f82073122a74268a464387659dd90bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:59.590303 2782 ts_tablet_manager.cc:902] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5: Initiating tablet copy from peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:59.634696 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:59.649194 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:59.672698 2796 ts_tablet_manager.cc:902] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:59.684787 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:59.685102 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:59.693593 2797 ts_tablet_manager.cc:902] T 43221a82a1124bf8840c6369f27f47f2 P 85f69eefe4294197923dff5efadcbbd5: Initiating tablet copy from peer e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:13:59.708101 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:59.735838 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:59.745457 2750 tablet_copy_client.cc:783] T 2f82073122a74268a464387659dd90bf P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:59.745752 2750 tablet_copy_client.cc:647] T 2f82073122a74268a464387659dd90bf P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:59.747761 507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:59.757442 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:59.792404 2750 tablet_copy_client.cc:523] T 2f82073122a74268a464387659dd90bf P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:59.812114 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:59.812390 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:59.828366 2782 tablet_copy_client.cc:308] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.68:37063
I20240503 14:13:59.829136 31352 tablet_copy_service.cc:140] P 82cfdc130877418dbe9d980d368973be: Received BeginTabletCopySession request for tablet 55cf34de74694f47ad3f9f4d9bdffae4 from peer 85f69eefe4294197923dff5efadcbbd5 ({username='slave'} at 127.22.242.72:36333)
I20240503 14:13:59.829326 31352 tablet_copy_service.cc:161] P 82cfdc130877418dbe9d980d368973be: Beginning new tablet copy session on tablet 55cf34de74694f47ad3f9f4d9bdffae4 from peer 85f69eefe4294197923dff5efadcbbd5 at {username='slave'} at 127.22.242.72:36333: session id = 85f69eefe4294197923dff5efadcbbd5-55cf34de74694f47ad3f9f4d9bdffae4
I20240503 14:13:59.833256 31352 tablet_copy_source_session.cc:205] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:59.834800 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:59.847577 2831 ts_tablet_manager.cc:902] T da9dd5cd81114817b0f898c6b8fb3c07 P 4665a3377574417ba0d3165c236533c0: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:13:59.848522 2827 ts_tablet_manager.cc:902] T 626fcb185af0446b89ae82066e08981d P 4665a3377574417ba0d3165c236533c0: Initiating tablet copy from peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:13:59.849156 2782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55cf34de74694f47ad3f9f4d9bdffae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:59.855181 2750 tablet_bootstrap.cc:492] T 2f82073122a74268a464387659dd90bf P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap starting.
I20240503 14:13:59.869345 507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:59.870258 2797 tablet_copy_client.cc:308] T 43221a82a1124bf8840c6369f27f47f2 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44297
I20240503 14:13:59.870927 31221 tablet_copy_service.cc:140] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Received BeginTabletCopySession request for tablet 43221a82a1124bf8840c6369f27f47f2 from peer 85f69eefe4294197923dff5efadcbbd5 ({username='slave'} at 127.22.242.72:40123)
I20240503 14:13:59.871124 31221 tablet_copy_service.cc:161] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Beginning new tablet copy session on tablet 43221a82a1124bf8840c6369f27f47f2 from peer 85f69eefe4294197923dff5efadcbbd5 at {username='slave'} at 127.22.242.72:40123: session id = 85f69eefe4294197923dff5efadcbbd5-43221a82a1124bf8840c6369f27f47f2
I20240503 14:13:59.874192 31221 tablet_copy_source_session.cc:205] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:59.879371 2797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43221a82a1124bf8840c6369f27f47f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:59.886818 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:59.887245 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:59.928390 2797 tablet_copy_client.cc:783] T 43221a82a1124bf8840c6369f27f47f2 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:59.928714 2797 tablet_copy_client.cc:647] T 43221a82a1124bf8840c6369f27f47f2 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:59.930516 2797 tablet_copy_client.cc:523] T 43221a82a1124bf8840c6369f27f47f2 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:59.934127 2797 tablet_bootstrap.cc:492] T 43221a82a1124bf8840c6369f27f47f2 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap starting.
I20240503 14:13:59.949462 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:59.954911 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:00.009114 2796 tablet_copy_client.cc:308] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:14:00.009812 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet ebd6dc710e144e61a5fae3911f25ee5c from peer 85f69eefe4294197923dff5efadcbbd5 ({username='slave'} at 127.22.242.72:54703)
I20240503 14:14:00.010416 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet ebd6dc710e144e61a5fae3911f25ee5c from peer 85f69eefe4294197923dff5efadcbbd5 at {username='slave'} at 127.22.242.72:54703: session id = 85f69eefe4294197923dff5efadcbbd5-ebd6dc710e144e61a5fae3911f25ee5c
I20240503 14:14:00.013265 31092 tablet_copy_source_session.cc:205] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:00.015928 2796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd6dc710e144e61a5fae3911f25ee5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:00.043929 2782 tablet_copy_client.cc:783] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:00.044261 2782 tablet_copy_client.cc:647] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:00.053951 2831 tablet_copy_client.cc:308] T da9dd5cd81114817b0f898c6b8fb3c07 P 4665a3377574417ba0d3165c236533c0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:14:00.060811 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet da9dd5cd81114817b0f898c6b8fb3c07 from peer 4665a3377574417ba0d3165c236533c0 ({username='slave'} at 127.22.242.71:41641)
I20240503 14:14:00.060984 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet da9dd5cd81114817b0f898c6b8fb3c07 from peer 4665a3377574417ba0d3165c236533c0 at {username='slave'} at 127.22.242.71:41641: session id = 4665a3377574417ba0d3165c236533c0-da9dd5cd81114817b0f898c6b8fb3c07
I20240503 14:14:00.063851 31092 tablet_copy_source_session.cc:205] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:00.064647 2079 raft_consensus.cc:1050] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER 6967bdda50404d68a5ac043b709440e1 to VOTER
I20240503 14:14:00.065424 2831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9dd5cd81114817b0f898c6b8fb3c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:00.065469 2079 consensus_queue.cc:237] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:14:00.088369 2827 tablet_copy_client.cc:308] T 626fcb185af0446b89ae82066e08981d P 4665a3377574417ba0d3165c236533c0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.68:37063
I20240503 14:14:00.090543 31352 tablet_copy_service.cc:140] P 82cfdc130877418dbe9d980d368973be: Received BeginTabletCopySession request for tablet 626fcb185af0446b89ae82066e08981d from peer 4665a3377574417ba0d3165c236533c0 ({username='slave'} at 127.22.242.71:44595)
I20240503 14:14:00.090728 31352 tablet_copy_service.cc:161] P 82cfdc130877418dbe9d980d368973be: Beginning new tablet copy session on tablet 626fcb185af0446b89ae82066e08981d from peer 4665a3377574417ba0d3165c236533c0 at {username='slave'} at 127.22.242.71:44595: session id = 4665a3377574417ba0d3165c236533c0-626fcb185af0446b89ae82066e08981d
I20240503 14:14:00.093887 31352 tablet_copy_source_session.cc:205] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:00.098956 2782 tablet_copy_client.cc:523] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:00.103727 2782 tablet_bootstrap.cc:492] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap starting.
I20240503 14:14:00.104548 2796 tablet_copy_client.cc:783] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:00.104794 2796 tablet_copy_client.cc:647] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:00.106053 2827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 626fcb185af0446b89ae82066e08981d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:00.106763 2796 tablet_copy_client.cc:523] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:00.112680 2831 tablet_copy_client.cc:783] T da9dd5cd81114817b0f898c6b8fb3c07 P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:00.112933 2831 tablet_copy_client.cc:647] T da9dd5cd81114817b0f898c6b8fb3c07 P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:00.113615 2796 tablet_bootstrap.cc:492] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5: Bootstrap starting.
I20240503 14:14:00.116895 393 raft_consensus.cc:1261] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:00.117517 2671 consensus_queue.cc:1035] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:00.125175 527 raft_consensus.cc:1261] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:00.125893 2671 consensus_queue.cc:1035] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:00.132323 2831 tablet_copy_client.cc:523] T da9dd5cd81114817b0f898c6b8fb3c07 P 4665a3377574417ba0d3165c236533c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:00.165305 31200 raft_consensus.cc:1261] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:00.165396 2827 tablet_copy_client.cc:783] T 626fcb185af0446b89ae82066e08981d P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:00.165776 2827 tablet_copy_client.cc:647] T 626fcb185af0446b89ae82066e08981d P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:00.166303 2671 consensus_queue.cc:1035] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:00.172190 2831 tablet_bootstrap.cc:492] T da9dd5cd81114817b0f898c6b8fb3c07 P 4665a3377574417ba0d3165c236533c0: Bootstrap starting.
I20240503 14:14:00.188657 2827 tablet_copy_client.cc:523] T 626fcb185af0446b89ae82066e08981d P 4665a3377574417ba0d3165c236533c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:00.193029 2827 tablet_bootstrap.cc:492] T 626fcb185af0446b89ae82066e08981d P 4665a3377574417ba0d3165c236533c0: Bootstrap starting.
I20240503 14:14:00.209810 2671 raft_consensus.cc:2932] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:00.216796 30793 catalog_manager.cc:5484] T c837def6d76242f08b20a448541e64d6 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 7 to 8, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:00.221515 528 raft_consensus.cc:2932] T c837def6d76242f08b20a448541e64d6 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:00.248994 31200 raft_consensus.cc:2932] T c837def6d76242f08b20a448541e64d6 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:00.274964 393 raft_consensus.cc:2932] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:00.348534 2831 tablet_bootstrap.cc:492] T da9dd5cd81114817b0f898c6b8fb3c07 P 4665a3377574417ba0d3165c236533c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:00.352123 2831 tablet_bootstrap.cc:492] T da9dd5cd81114817b0f898c6b8fb3c07 P 4665a3377574417ba0d3165c236533c0: Bootstrap complete.
I20240503 14:14:00.352453 2831 ts_tablet_manager.cc:1367] T da9dd5cd81114817b0f898c6b8fb3c07 P 4665a3377574417ba0d3165c236533c0: Time spent bootstrapping tablet: real 0.180s user 0.047s sys 0.006s
I20240503 14:14:00.353442 2831 raft_consensus.cc:348] T da9dd5cd81114817b0f898c6b8fb3c07 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:14:00.353726 2831 raft_consensus.cc:729] T da9dd5cd81114817b0f898c6b8fb3c07 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4665a3377574417ba0d3165c236533c0, State: Initialized, Role: LEARNER
I20240503 14:14:00.356205 2831 consensus_queue.cc:260] T da9dd5cd81114817b0f898c6b8fb3c07 P 4665a3377574417ba0d3165c236533c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:14:00.368368 2831 ts_tablet_manager.cc:1398] T da9dd5cd81114817b0f898c6b8fb3c07 P 4665a3377574417ba0d3165c236533c0: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20240503 14:14:00.370256 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session 4665a3377574417ba0d3165c236533c0-da9dd5cd81114817b0f898c6b8fb3c07 received from {username='slave'} at 127.22.242.71:41641
I20240503 14:14:00.370404 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session 4665a3377574417ba0d3165c236533c0-da9dd5cd81114817b0f898c6b8fb3c07 on tablet da9dd5cd81114817b0f898c6b8fb3c07 with peer 4665a3377574417ba0d3165c236533c0
I20240503 14:14:00.370846 660 raft_consensus.cc:1203] T da9dd5cd81114817b0f898c6b8fb3c07 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:00.379458 2750 tablet_bootstrap.cc:492] T 2f82073122a74268a464387659dd90bf P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:00.380901 2750 tablet_bootstrap.cc:492] T 2f82073122a74268a464387659dd90bf P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap complete.
I20240503 14:14:00.381234 2750 ts_tablet_manager.cc:1367] T 2f82073122a74268a464387659dd90bf P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent bootstrapping tablet: real 0.526s user 0.059s sys 0.001s
I20240503 14:14:00.382006 2750 raft_consensus.cc:348] T 2f82073122a74268a464387659dd90bf P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:14:00.382277 2750 raft_consensus.cc:729] T 2f82073122a74268a464387659dd90bf P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa71cbdf0e544b81bd4e7e27196d2c91, State: Initialized, Role: LEARNER
I20240503 14:14:00.382488 2750 consensus_queue.cc:260] T 2f82073122a74268a464387659dd90bf P fa71cbdf0e544b81bd4e7e27196d2c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:14:00.383087 2750 ts_tablet_manager.cc:1398] T 2f82073122a74268a464387659dd90bf P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:00.383805 2827 tablet_bootstrap.cc:492] T 626fcb185af0446b89ae82066e08981d P 4665a3377574417ba0d3165c236533c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:00.383843 31221 tablet_copy_service.cc:342] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Request end of tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-2f82073122a74268a464387659dd90bf received from {username='slave'} at 127.22.242.70:33209
I20240503 14:14:00.383989 31221 tablet_copy_service.cc:433] P e7badbe63a8a4d9ca0c78818c6f4c8ce: ending tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-2f82073122a74268a464387659dd90bf on tablet 2f82073122a74268a464387659dd90bf with peer fa71cbdf0e544b81bd4e7e27196d2c91
I20240503 14:14:00.384341 2827 tablet_bootstrap.cc:492] T 626fcb185af0446b89ae82066e08981d P 4665a3377574417ba0d3165c236533c0: Bootstrap complete.
I20240503 14:14:00.384639 2827 ts_tablet_manager.cc:1367] T 626fcb185af0446b89ae82066e08981d P 4665a3377574417ba0d3165c236533c0: Time spent bootstrapping tablet: real 0.192s user 0.058s sys 0.011s
I20240503 14:14:00.385591 2827 raft_consensus.cc:348] T 626fcb185af0446b89ae82066e08981d P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:14:00.385919 2827 raft_consensus.cc:729] T 626fcb185af0446b89ae82066e08981d P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4665a3377574417ba0d3165c236533c0, State: Initialized, Role: LEARNER
I20240503 14:14:00.386211 2827 consensus_queue.cc:260] T 626fcb185af0446b89ae82066e08981d P 4665a3377574417ba0d3165c236533c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:14:00.386947 2827 ts_tablet_manager.cc:1398] T 626fcb185af0446b89ae82066e08981d P 4665a3377574417ba0d3165c236533c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:00.387784 31352 tablet_copy_service.cc:342] P 82cfdc130877418dbe9d980d368973be: Request end of tablet copy session 4665a3377574417ba0d3165c236533c0-626fcb185af0446b89ae82066e08981d received from {username='slave'} at 127.22.242.71:44595
I20240503 14:14:00.387941 31352 tablet_copy_service.cc:433] P 82cfdc130877418dbe9d980d368973be: ending tablet copy session 4665a3377574417ba0d3165c236533c0-626fcb185af0446b89ae82066e08981d on tablet 626fcb185af0446b89ae82066e08981d with peer 4665a3377574417ba0d3165c236533c0
I20240503 14:14:00.454689 2797 tablet_bootstrap.cc:492] T 43221a82a1124bf8840c6369f27f47f2 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:00.455227 2797 tablet_bootstrap.cc:492] T 43221a82a1124bf8840c6369f27f47f2 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap complete.
I20240503 14:14:00.455515 2797 ts_tablet_manager.cc:1367] T 43221a82a1124bf8840c6369f27f47f2 P 85f69eefe4294197923dff5efadcbbd5: Time spent bootstrapping tablet: real 0.521s user 0.049s sys 0.009s
I20240503 14:14:00.461845 528 raft_consensus.cc:1203] T 2f82073122a74268a464387659dd90bf P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 14:14:00.467530 2797 raft_consensus.cc:348] T 43221a82a1124bf8840c6369f27f47f2 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:00.493515 31072 consensus_queue.cc:237] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:00.506166 31200 raft_consensus.cc:1261] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:00.507094 30936 raft_consensus.cc:1261] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:00.508814 2079 consensus_queue.cc:1035] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:00.510296 2855 consensus_queue.cc:1035] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:00.514122 2855 raft_consensus.cc:2932] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:00.514680 31199 raft_consensus.cc:2932] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:00.522367 30791 catalog_manager.cc:5484] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
W20240503 14:14:00.540171 31008 consensus_peers.cc:485] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6b9a10c474f46bda0310ce661022b42. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:00.551044 2782 tablet_bootstrap.cc:492] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:00.551631 2782 tablet_bootstrap.cc:492] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap complete.
I20240503 14:14:00.551949 2782 ts_tablet_manager.cc:1367] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5: Time spent bootstrapping tablet: real 0.448s user 0.050s sys 0.010s
I20240503 14:14:00.552861 2782 raft_consensus.cc:348] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:00.553222 2782 raft_consensus.cc:729] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85f69eefe4294197923dff5efadcbbd5, State: Initialized, Role: LEARNER
I20240503 14:14:00.553491 2782 consensus_queue.cc:260] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:00.559780 2797 raft_consensus.cc:729] T 43221a82a1124bf8840c6369f27f47f2 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85f69eefe4294197923dff5efadcbbd5, State: Initialized, Role: LEARNER
I20240503 14:14:00.565800 30936 raft_consensus.cc:2932] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:00.568070 2797 consensus_queue.cc:260] T 43221a82a1124bf8840c6369f27f47f2 P 85f69eefe4294197923dff5efadcbbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:00.572120 2782 ts_tablet_manager.cc:1398] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5: Time spent starting tablet: real 0.020s user 0.001s sys 0.005s
I20240503 14:14:00.573017 31352 tablet_copy_service.cc:342] P 82cfdc130877418dbe9d980d368973be: Request end of tablet copy session 85f69eefe4294197923dff5efadcbbd5-55cf34de74694f47ad3f9f4d9bdffae4 received from {username='slave'} at 127.22.242.72:36333
I20240503 14:14:00.573210 31352 tablet_copy_service.cc:433] P 82cfdc130877418dbe9d980d368973be: ending tablet copy session 85f69eefe4294197923dff5efadcbbd5-55cf34de74694f47ad3f9f4d9bdffae4 on tablet 55cf34de74694f47ad3f9f4d9bdffae4 with peer 85f69eefe4294197923dff5efadcbbd5
I20240503 14:14:00.573454 2797 ts_tablet_manager.cc:1398] T 43221a82a1124bf8840c6369f27f47f2 P 85f69eefe4294197923dff5efadcbbd5: Time spent starting tablet: real 0.117s user 0.004s sys 0.000s
I20240503 14:14:00.574952 31221 tablet_copy_service.cc:342] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Request end of tablet copy session 85f69eefe4294197923dff5efadcbbd5-43221a82a1124bf8840c6369f27f47f2 received from {username='slave'} at 127.22.242.72:40123
I20240503 14:14:00.575107 31221 tablet_copy_service.cc:433] P e7badbe63a8a4d9ca0c78818c6f4c8ce: ending tablet copy session 85f69eefe4294197923dff5efadcbbd5-43221a82a1124bf8840c6369f27f47f2 on tablet 43221a82a1124bf8840c6369f27f47f2 with peer 85f69eefe4294197923dff5efadcbbd5
I20240503 14:14:00.576479 792 raft_consensus.cc:1203] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:00.594990 2796 tablet_bootstrap.cc:492] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:00.596016 2796 tablet_bootstrap.cc:492] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5: Bootstrap complete.
I20240503 14:14:00.596676 2796 ts_tablet_manager.cc:1367] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5: Time spent bootstrapping tablet: real 0.483s user 0.061s sys 0.019s
I20240503 14:14:00.597787 2796 raft_consensus.cc:348] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:00.598130 2796 raft_consensus.cc:729] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85f69eefe4294197923dff5efadcbbd5, State: Initialized, Role: LEARNER
I20240503 14:14:00.598517 2796 consensus_queue.cc:260] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:00.600054 2796 ts_tablet_manager.cc:1398] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:14:00.601049 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session 85f69eefe4294197923dff5efadcbbd5-ebd6dc710e144e61a5fae3911f25ee5c received from {username='slave'} at 127.22.242.72:54703
I20240503 14:14:00.601214 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session 85f69eefe4294197923dff5efadcbbd5-ebd6dc710e144e61a5fae3911f25ee5c on tablet ebd6dc710e144e61a5fae3911f25ee5c with peer 85f69eefe4294197923dff5efadcbbd5
I20240503 14:14:00.609526 31072 consensus_queue.cc:237] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:00.625072 792 raft_consensus.cc:1203] T 43221a82a1124bf8840c6369f27f47f2 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:00.626291 393 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:00.626435 31330 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:00.627893 2446 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:00.628623 2446 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 14:14:00.633363 31008 consensus_peers.cc:485] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:00.645440 2446 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:00.648928 31326 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:00.658663 30791 catalog_manager.cc:5484] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index 8 to 9, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:00.673655 31072 consensus_queue.cc:237] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:14:00.680460 393 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:00.691479 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:00.730254 792 raft_consensus.cc:1261] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:00.730912 2855 consensus_queue.cc:1035] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:14:00.734781 31008 consensus_peers.cc:485] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3e80559017d4a6ba16d34e0b7aab085. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:00.739660 31330 raft_consensus.cc:1261] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:00.740311 2671 consensus_queue.cc:1035] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:00.749126 2855 raft_consensus.cc:2932] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:00.753832 792 raft_consensus.cc:2932] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:00.762315 660 raft_consensus.cc:1203] T 626fcb185af0446b89ae82066e08981d P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 14:14:00.768031 30792 catalog_manager.cc:5484] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 reported cstate change: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:00.795737 31326 raft_consensus.cc:2932] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:00.813266 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:00.847855 30936 consensus_queue.cc:237] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:14:00.852870 2090 raft_consensus.cc:1050] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce: attempting to promote NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 to VOTER
I20240503 14:14:00.853950 2090 consensus_queue.cc:237] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:14:00.883106 2671 raft_consensus.cc:1050] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER 4665a3377574417ba0d3165c236533c0 to VOTER
I20240503 14:14:00.884233 2671 consensus_queue.cc:237] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:14:00.891038 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:00.897025 660 raft_consensus.cc:1261] T da9dd5cd81114817b0f898c6b8fb3c07 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:00.897609 2446 consensus_queue.cc:1035] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:14:00.898386 30874 consensus_peers.cc:485] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 -> Peer fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829): Couldn't send request to peer fa71cbdf0e544b81bd4e7e27196d2c91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6aeaa3fe1cd4127a6144d85b0434057. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:00.900269 394 raft_consensus.cc:1261] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:00.901681 2633 consensus_queue.cc:1035] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:00.903620 31330 raft_consensus.cc:1261] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:00.904130 2446 consensus_queue.cc:1035] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:00.909008 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:14:00.909240 31072 raft_consensus.cc:1261] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:00.909792 2633 consensus_queue.cc:1035] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:00.920840 30936 raft_consensus.cc:1261] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:00.921525 2671 consensus_queue.cc:1035] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:00.936431 2796 ts_tablet_manager.cc:902] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:14:00.936403 507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:00.952936 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:00.958858 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:14:00.980559 2979 log.cc:924] Time spent T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20240503 14:14:00.988345 2446 raft_consensus.cc:2932] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:14:00.993503 2991 raft_consensus.cc:2932] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:14:00.993947 31326 raft_consensus.cc:2932] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:14:00.998922 659 raft_consensus.cc:2932] T da9dd5cd81114817b0f898c6b8fb3c07 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:14:01.001264 30793 catalog_manager.cc:5484] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:01.013583 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:14:01.014523 30877 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.66:33221, user_credentials={real_user=slave}} blocked reactor thread for 45116.4us
I20240503 14:14:01.013465 30936 raft_consensus.cc:2932] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:14:01.025668 31200 consensus_queue.cc:237] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:14:01.029000 30792 catalog_manager.cc:5484] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 6 to 7, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "0dc902d91ccc435a82a5ede4a7729811" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:01.034181 31072 raft_consensus.cc:2932] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:14:01.041402 393 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:14:01.042138 2753 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:01.048406 394 raft_consensus.cc:2932] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:14:01.050673 528 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:14:01.051391 2753 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:01.059821 2796 tablet_copy_client.cc:308] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:14:01.060673 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet d6b9a10c474f46bda0310ce661022b42 from peer 85f69eefe4294197923dff5efadcbbd5 ({username='slave'} at 127.22.242.72:54703)
I20240503 14:14:01.060883 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet d6b9a10c474f46bda0310ce661022b42 from peer 85f69eefe4294197923dff5efadcbbd5 at {username='slave'} at 127.22.242.72:54703: session id = 85f69eefe4294197923dff5efadcbbd5-d6b9a10c474f46bda0310ce661022b42
I20240503 14:14:01.065352 31092 tablet_copy_source_session.cc:205] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:01.066256 31326 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:14:01.067001 2796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b9a10c474f46bda0310ce661022b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:01.067101 2753 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:01.076352 2639 raft_consensus.cc:1050] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce: attempting to promote NON_VOTER 85f69eefe4294197923dff5efadcbbd5 to VOTER
I20240503 14:14:01.077431 2639 consensus_queue.cc:237] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:01.097359 2753 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:14:01.098340 528 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:14:01.104446 31326 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:14:01.105296 30800 catalog_manager.cc:5484] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 9 to 10, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:01.106953 393 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:14:01.110842 2796 tablet_copy_client.cc:783] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:01.111198 2796 tablet_copy_client.cc:647] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 1 WAL segments...
W20240503 14:14:01.111809 30875 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.67:44297, user_credentials={real_user=slave}} blocked reactor thread for 102694us
I20240503 14:14:01.113137 2685 raft_consensus.cc:1050] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be: attempting to promote NON_VOTER 85f69eefe4294197923dff5efadcbbd5 to VOTER
I20240503 14:14:01.114109 2685 consensus_queue.cc:237] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:01.116500 31199 consensus_queue.cc:237] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:14:01.122555 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:01.124356 526 raft_consensus.cc:1261] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:01.125113 791 raft_consensus.cc:1261] T 43221a82a1124bf8840c6369f27f47f2 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:01.128528 30938 raft_consensus.cc:1261] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:01.128453 2090 consensus_queue.cc:1035] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:01.130790 525 raft_consensus.cc:1261] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:01.133953 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:01.136438 2753 consensus_queue.cc:1035] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:01.143848 2090 consensus_queue.cc:1035] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:01.148532 30936 raft_consensus.cc:1261] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:01.154664 791 raft_consensus.cc:1203] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:01.160882 2090 consensus_queue.cc:1035] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:01.167630 2796 tablet_copy_client.cc:523] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:01.176532 2639 consensus_queue.cc:1035] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:01.178694 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:14:01.184859 2753 raft_consensus.cc:2932] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:01.186100 526 raft_consensus.cc:2932] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:01.195612 394 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:14:01.208235 2639 raft_consensus.cc:2932] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:01.218263 527 raft_consensus.cc:1261] T 2f82073122a74268a464387659dd90bf P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:14:01.220522 30936 raft_consensus.cc:2932] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:01.233175 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:01.236230 30792 catalog_manager.cc:5484] T 43221a82a1124bf8840c6369f27f47f2 P fa71cbdf0e544b81bd4e7e27196d2c91 reported cstate change: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:01.246773 2090 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:14:01.248800 3002 consensus_queue.cc:1035] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:14:01.249905 791 raft_consensus.cc:2932] T 43221a82a1124bf8840c6369f27f47f2 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:01.266088 30792 catalog_manager.cc:5484] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:01.283816 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:14:01.283929 31138 consensus_peers.cc:485] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91a7b8a53140429b956178d43478a541. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:01.285655 3002 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:14:01.289667 526 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:14:01.294068 30936 raft_consensus.cc:2932] T 43221a82a1124bf8840c6369f27f47f2 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:01.295272 30785 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2f82073122a74268a464387659dd90bf with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:01.297735 393 raft_consensus.cc:2932] T 2f82073122a74268a464387659dd90bf P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:14:01.301793 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:01.301529 30792 catalog_manager.cc:5484] T 2f82073122a74268a464387659dd90bf P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 10 to 11, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:01.316993 3025 ts_tablet_manager.cc:902] T f3e80559017d4a6ba16d34e0b7aab085 P 4665a3377574417ba0d3165c236533c0: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:14:01.317864 3003 ts_tablet_manager.cc:902] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:14:01.321854 31311 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f82073122a74268a464387659dd90bf with delete_type TABLET_DATA_TOMBSTONED (TS 82cfdc130877418dbe9d980d368973be not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:48102
I20240503 14:14:01.323184 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:01.335361 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:01.347828 526 raft_consensus.cc:2932] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:01.355486 660 raft_consensus.cc:1261] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:01.357110 792 raft_consensus.cc:1261] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:01.359314 31312 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:01.388775 2707 consensus_queue.cc:1035] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:01.389281 3038 tablet_replica.cc:330] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be: stopping tablet replica
I20240503 14:14:01.389555 3038 raft_consensus.cc:2229] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:01.389875 3038 raft_consensus.cc:2258] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:01.391315 3038 ts_tablet_manager.cc:1875] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:01.389515 2707 consensus_queue.cc:1035] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20240503 14:14:01.404011 2796 tablet_bootstrap.cc:492] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap starting.
I20240503 14:14:01.424032 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:01.434022 31072 consensus_queue.cc:237] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
W20240503 14:14:01.437466 31268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.71:46453, user_credentials={real_user=slave}} blocked reactor thread for 76793.5us
W20240503 14:14:01.438877 31008 consensus_peers.cc:485] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189): Couldn't send request to peer 6967bdda50404d68a5ac043b709440e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 584769402a7e4e8a985a8f2363e2b8b9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:01.439402 526 raft_consensus.cc:1261] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:01.440006 2079 consensus_queue.cc:1035] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:01.444468 31330 raft_consensus.cc:1261] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:01.445578 2671 consensus_queue.cc:1035] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:01.449667 2671 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:14:01.452963 31326 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:14:01.461498 30792 catalog_manager.cc:5484] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 6 to 7, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:01.467761 525 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:14:01.478803 3033 raft_consensus.cc:1050] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be: attempting to promote NON_VOTER 4665a3377574417ba0d3165c236533c0 to VOTER
I20240503 14:14:01.479897 3033 consensus_queue.cc:237] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:14:01.497164 3056 ts_tablet_manager.cc:902] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91: Initiating tablet copy from peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:14:01.497254 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:14:01.497797 3025 tablet_copy_client.cc:308] T f3e80559017d4a6ba16d34e0b7aab085 P 4665a3377574417ba0d3165c236533c0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:14:01.500593 30938 raft_consensus.cc:1261] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:01.500869 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet f3e80559017d4a6ba16d34e0b7aab085 from peer 4665a3377574417ba0d3165c236533c0 ({username='slave'} at 127.22.242.71:41641)
I20240503 14:14:01.501040 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet f3e80559017d4a6ba16d34e0b7aab085 from peer 4665a3377574417ba0d3165c236533c0 at {username='slave'} at 127.22.242.71:41641: session id = 4665a3377574417ba0d3165c236533c0-f3e80559017d4a6ba16d34e0b7aab085
I20240503 14:14:01.501240 3031 consensus_queue.cc:1035] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:01.506151 31092 tablet_copy_source_session.cc:205] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:01.507709 3025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e80559017d4a6ba16d34e0b7aab085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:01.521435 3056 tablet_copy_client.cc:308] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:46013
I20240503 14:14:01.523828 2796 tablet_bootstrap.cc:700] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5: Time spent opening tablet: real 0.118s user 0.000s sys 0.001s
I20240503 14:14:01.528430 30961 tablet_copy_service.cc:140] P 0dc902d91ccc435a82a5ede4a7729811: Received BeginTabletCopySession request for tablet f6aeaa3fe1cd4127a6144d85b0434057 from peer fa71cbdf0e544b81bd4e7e27196d2c91 ({username='slave'} at 127.22.242.70:54795)
I20240503 14:14:01.528613 30961 tablet_copy_service.cc:161] P 0dc902d91ccc435a82a5ede4a7729811: Beginning new tablet copy session on tablet f6aeaa3fe1cd4127a6144d85b0434057 from peer fa71cbdf0e544b81bd4e7e27196d2c91 at {username='slave'} at 127.22.242.70:54795: session id = fa71cbdf0e544b81bd4e7e27196d2c91-f6aeaa3fe1cd4127a6144d85b0434057
I20240503 14:14:01.528920 792 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:01.529536 3025 tablet_copy_client.cc:783] T f3e80559017d4a6ba16d34e0b7aab085 P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:01.529820 3025 tablet_copy_client.cc:647] T f3e80559017d4a6ba16d34e0b7aab085 P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:01.531778 30961 tablet_copy_source_session.cc:205] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:01.528141 3031 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:01.538589 507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:01.540570 30936 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:01.546955 30792 catalog_manager.cc:5484] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:01.548739 3038 ts_tablet_manager.cc:1888] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 14:14:01.548939 3038 log.cc:1196] T 2f82073122a74268a464387659dd90bf P 82cfdc130877418dbe9d980d368973be: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/wals/2f82073122a74268a464387659dd90bf
I20240503 14:14:01.549803 3056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6aeaa3fe1cd4127a6144d85b0434057. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:01.549973 30786 catalog_manager.cc:4830] TS 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063): tablet 2f82073122a74268a464387659dd90bf (table rebalance_test_table_0 [id=fa8e781fa3e140b6865ce5f7b429e4c4]) successfully deleted
I20240503 14:14:01.565387 659 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:01.568100 31326 consensus_queue.cc:237] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:01.572465 792 raft_consensus.cc:1261] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:01.573212 3031 consensus_queue.cc:1035] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:01.574628 3025 tablet_copy_client.cc:523] T f3e80559017d4a6ba16d34e0b7aab085 P 4665a3377574417ba0d3165c236533c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:01.574667 3069 ts_tablet_manager.cc:902] T 91a7b8a53140429b956178d43478a541 P 4665a3377574417ba0d3165c236533c0: Initiating tablet copy from peer e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:14:01.576815 3042 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:01.577409 792 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:01.583048 30786 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 55cf34de74694f47ad3f9f4d9bdffae4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:01.585680 30792 catalog_manager.cc:5484] T 55cf34de74694f47ad3f9f4d9bdffae4 P 85f69eefe4294197923dff5efadcbbd5 reported cstate change: config changed from index 8 to 9, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:01.590294 30920 tablet_service.cc:1514] Processing DeleteTablet for tablet 55cf34de74694f47ad3f9f4d9bdffae4 with delete_type TABLET_DATA_TOMBSTONED (TS 0dc902d91ccc435a82a5ede4a7729811 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56642
I20240503 14:14:01.592527 3003 tablet_copy_client.cc:308] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:14:01.593263 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet 63b2120788a54af78f02449f46bf23c7 from peer 85f69eefe4294197923dff5efadcbbd5 ({username='slave'} at 127.22.242.72:54703)
I20240503 14:14:01.593489 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet 63b2120788a54af78f02449f46bf23c7 from peer 85f69eefe4294197923dff5efadcbbd5 at {username='slave'} at 127.22.242.72:54703: session id = 85f69eefe4294197923dff5efadcbbd5-63b2120788a54af78f02449f46bf23c7
I20240503 14:14:01.601308 31092 tablet_copy_source_session.cc:205] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:01.602821 3003 ts_tablet_manager.cc:1875] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:14:01.607245 3072 tablet_replica.cc:330] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811: stopping tablet replica
I20240503 14:14:01.607471 3072 raft_consensus.cc:2229] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:01.607761 3072 raft_consensus.cc:2258] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:01.610322 3072 ts_tablet_manager.cc:1875] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:01.626865 3003 ts_tablet_manager.cc:1888] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 14:14:01.630874 3056 tablet_copy_client.cc:783] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:01.631139 3056 tablet_copy_client.cc:647] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:01.626454 3025 tablet_bootstrap.cc:492] T f3e80559017d4a6ba16d34e0b7aab085 P 4665a3377574417ba0d3165c236533c0: Bootstrap starting.
I20240503 14:14:01.631829 3003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b2120788a54af78f02449f46bf23c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:01.635526 658 raft_consensus.cc:1261] T 626fcb185af0446b89ae82066e08981d P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:14:01.635668 791 raft_consensus.cc:1261] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:14:01.636317 31072 raft_consensus.cc:1261] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:14:01.636317 2685 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:01.636744 3042 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:01.637123 3056 tablet_copy_client.cc:523] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:01.638890 3042 consensus_queue.cc:1035] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:01.640542 3072 ts_tablet_manager.cc:1888] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:01.642429 2079 raft_consensus.cc:1050] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER 85f69eefe4294197923dff5efadcbbd5 to VOTER
I20240503 14:14:01.643427 2079 consensus_queue.cc:237] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:01.644919 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:01.646550 3056 tablet_bootstrap.cc:492] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap starting.
I20240503 14:14:01.649886 3072 log.cc:1196] T 55cf34de74694f47ad3f9f4d9bdffae4 P 0dc902d91ccc435a82a5ede4a7729811: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/55cf34de74694f47ad3f9f4d9bdffae4
I20240503 14:14:01.651517 30784 catalog_manager.cc:4830] TS 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): tablet 55cf34de74694f47ad3f9f4d9bdffae4 (table rebalance_test_table_5 [id=04f8dfb749c04486895d6b0f7c206446]) successfully deleted
I20240503 14:14:01.664392 3003 tablet_copy_client.cc:783] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:01.664652 3003 tablet_copy_client.cc:647] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:01.665397 3042 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:14:01.667914 3003 tablet_copy_client.cc:523] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:01.672399 30800 catalog_manager.cc:5484] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index 9 to 10, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:01.673035 658 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:14:01.678313 31072 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:14:01.679162 792 raft_consensus.cc:1261] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:01.679802 2671 consensus_queue.cc:1035] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:01.681354 791 raft_consensus.cc:2932] T 626fcb185af0446b89ae82066e08981d P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:14:01.690682 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:01.704860 657 raft_consensus.cc:1261] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:01.705668 2671 consensus_queue.cc:1035] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:01.715827 30938 raft_consensus.cc:1261] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:01.716742 2671 consensus_queue.cc:1035] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:01.722487 660 raft_consensus.cc:1261] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:01.724057 3033 consensus_queue.cc:1035] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:01.726404 659 raft_consensus.cc:2932] T 55cf34de74694f47ad3f9f4d9bdffae4 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:01.743280 31071 consensus_queue.cc:237] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:01.765069 507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:01.765926 3003 tablet_bootstrap.cc:492] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap starting.
I20240503 14:14:01.781280 2671 raft_consensus.cc:2932] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:01.788727 659 raft_consensus.cc:2932] T ebd6dc710e144e61a5fae3911f25ee5c P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:01.790499 30800 catalog_manager.cc:5484] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:01.791407 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:01.795063 791 raft_consensus.cc:2932] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:01.822249 3069 tablet_copy_client.cc:308] T 91a7b8a53140429b956178d43478a541 P 4665a3377574417ba0d3165c236533c0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44297
W20240503 14:14:01.847026 31005 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.70:46829, user_credentials={real_user=slave}} blocked reactor thread for 48199.2us
I20240503 14:14:01.849365 30938 raft_consensus.cc:2932] T ebd6dc710e144e61a5fae3911f25ee5c P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
W20240503 14:14:01.853324 31008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.72:36545, user_credentials={real_user=slave}} blocked reactor thread for 52486.4us
I20240503 14:14:01.854472 393 raft_consensus.cc:1261] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 14:14:01.859932 31008 consensus_peers.cc:485] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c86ee72320b647eb84e376f8995b55d2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:01.860769 2855 consensus_queue.cc:1035] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:01.860145 31200 raft_consensus.cc:1261] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:01.861541 3091 consensus_queue.cc:1035] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:01.892460 31199 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:01.897516 2079 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:01.898295 30800 catalog_manager.cc:5484] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:01.910419 31221 tablet_copy_service.cc:140] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Received BeginTabletCopySession request for tablet 91a7b8a53140429b956178d43478a541 from peer 4665a3377574417ba0d3165c236533c0 ({username='slave'} at 127.22.242.71:44441)
I20240503 14:14:01.910673 31221 tablet_copy_service.cc:161] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Beginning new tablet copy session on tablet 91a7b8a53140429b956178d43478a541 from peer 4665a3377574417ba0d3165c236533c0 at {username='slave'} at 127.22.242.71:44441: session id = 4665a3377574417ba0d3165c236533c0-91a7b8a53140429b956178d43478a541
I20240503 14:14:01.914788 31221 tablet_copy_source_session.cc:205] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:01.917090 3069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a7b8a53140429b956178d43478a541. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:01.927342 3069 tablet_copy_client.cc:783] T 91a7b8a53140429b956178d43478a541 P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:01.927628 3069 tablet_copy_client.cc:647] T 91a7b8a53140429b956178d43478a541 P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:01.929603 3069 tablet_copy_client.cc:523] T 91a7b8a53140429b956178d43478a541 P 4665a3377574417ba0d3165c236533c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:01.932950 394 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:01.939107 3069 tablet_bootstrap.cc:492] T 91a7b8a53140429b956178d43478a541 P 4665a3377574417ba0d3165c236533c0: Bootstrap starting.
I20240503 14:14:02.001793 31071 consensus_queue.cc:237] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:14:02.030836 3111 ts_tablet_manager.cc:902] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:14:02.042881 31326 raft_consensus.cc:1261] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:02.043870 3092 consensus_queue.cc:1035] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20240503 14:14:02.046281 31008 consensus_peers.cc:485] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7b5dd5eb46a45989bf9ffe3144bc189. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:02.057878 526 raft_consensus.cc:1261] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:02.058509 3099 consensus_queue.cc:1035] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:02.063130 31200 consensus_queue.cc:237] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:14:02.078483 3092 raft_consensus.cc:2932] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:02.080986 393 raft_consensus.cc:1261] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:02.081631 30938 raft_consensus.cc:1261] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:02.082726 31071 raft_consensus.cc:1261] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:02.084319 3002 consensus_queue.cc:1035] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:02.085211 3002 consensus_queue.cc:1035] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:02.085748 2090 consensus_queue.cc:1035] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:02.087797 526 raft_consensus.cc:2932] T e7b5dd5eb46a45989bf9ffe3144bc189 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:02.093103 30800 catalog_manager.cc:5484] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:02.107931 3003 tablet_bootstrap.cc:492] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:02.108634 3003 tablet_bootstrap.cc:492] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap complete.
I20240503 14:14:02.109057 3003 ts_tablet_manager.cc:1367] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: Time spent bootstrapping tablet: real 0.343s user 0.062s sys 0.000s
I20240503 14:14:02.109925 3003 raft_consensus.cc:348] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:02.110322 3003 raft_consensus.cc:729] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85f69eefe4294197923dff5efadcbbd5, State: Initialized, Role: LEARNER
I20240503 14:14:02.110566 3003 consensus_queue.cc:260] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:02.112105 3003 ts_tablet_manager.cc:1398] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:02.112936 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session 85f69eefe4294197923dff5efadcbbd5-63b2120788a54af78f02449f46bf23c7 received from {username='slave'} at 127.22.242.72:54703
I20240503 14:14:02.113088 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session 85f69eefe4294197923dff5efadcbbd5-63b2120788a54af78f02449f46bf23c7 on tablet 63b2120788a54af78f02449f46bf23c7 with peer 85f69eefe4294197923dff5efadcbbd5
I20240503 14:14:02.138283 31326 raft_consensus.cc:2932] T e7b5dd5eb46a45989bf9ffe3144bc189 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
W20240503 14:14:02.160698 31138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.66:33221, user_credentials={real_user=slave}} blocked reactor thread for 77273.4us
I20240503 14:14:02.183633 3128 raft_consensus.cc:2932] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:02.188954 30938 raft_consensus.cc:2932] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:02.193506 30800 catalog_manager.cc:5484] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:02.196375 393 raft_consensus.cc:2932] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:02.212910 31200 consensus_queue.cc:237] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
W20240503 14:14:02.217921 31138 consensus_peers.cc:485] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3330e28c2d7434699b78070dd92ed73. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:02.221875 30936 raft_consensus.cc:1261] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:02.222723 3128 consensus_queue.cc:1035] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:02.224488 3111 tablet_copy_client.cc:308] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:14:02.225171 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet 584769402a7e4e8a985a8f2363e2b8b9 from peer 6967bdda50404d68a5ac043b709440e1 ({username='slave'} at 127.22.242.69:44021)
I20240503 14:14:02.225873 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet 584769402a7e4e8a985a8f2363e2b8b9 from peer 6967bdda50404d68a5ac043b709440e1 at {username='slave'} at 127.22.242.69:44021: session id = 6967bdda50404d68a5ac043b709440e1-584769402a7e4e8a985a8f2363e2b8b9
I20240503 14:14:02.229715 31072 raft_consensus.cc:1261] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:02.230499 3025 tablet_bootstrap.cc:492] T f3e80559017d4a6ba16d34e0b7aab085 P 4665a3377574417ba0d3165c236533c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:02.230712 3002 consensus_queue.cc:1035] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:02.230944 3025 tablet_bootstrap.cc:492] T f3e80559017d4a6ba16d34e0b7aab085 P 4665a3377574417ba0d3165c236533c0: Bootstrap complete.
I20240503 14:14:02.231251 3025 ts_tablet_manager.cc:1367] T f3e80559017d4a6ba16d34e0b7aab085 P 4665a3377574417ba0d3165c236533c0: Time spent bootstrapping tablet: real 0.605s user 0.049s sys 0.004s
I20240503 14:14:02.231377 31092 tablet_copy_source_session.cc:205] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:02.233152 394 raft_consensus.cc:1261] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:02.232839 31071 raft_consensus.cc:2932] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:02.234383 3002 consensus_queue.cc:1035] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:02.237195 3111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584769402a7e4e8a985a8f2363e2b8b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:02.237473 3128 raft_consensus.cc:2932] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:02.242005 30785 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f3330e28c2d7434699b78070dd92ed73 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:14:02.244755 394 raft_consensus.cc:2932] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:02.245371 30800 catalog_manager.cc:5484] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:02.243639 30938 raft_consensus.cc:2932] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:02.264245 3111 tablet_copy_client.cc:783] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:02.264525 3111 tablet_copy_client.cc:647] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:02.267796 3111 tablet_copy_client.cc:523] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:02.274406 31072 raft_consensus.cc:2932] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:02.277004 31200 consensus_queue.cc:237] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:14:02.278970 3111 tablet_bootstrap.cc:492] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1: Bootstrap starting.
I20240503 14:14:02.282253 30936 raft_consensus.cc:1261] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:02.283047 2090 consensus_queue.cc:1035] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:02.287272 2090 raft_consensus.cc:2932] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:02.291324 30936 raft_consensus.cc:2932] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:02.232034 3025 raft_consensus.cc:348] T f3e80559017d4a6ba16d34e0b7aab085 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:14:02.298074 3025 raft_consensus.cc:729] T f3e80559017d4a6ba16d34e0b7aab085 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4665a3377574417ba0d3165c236533c0, State: Initialized, Role: LEARNER
I20240503 14:14:02.298339 3025 consensus_queue.cc:260] T f3e80559017d4a6ba16d34e0b7aab085 P 4665a3377574417ba0d3165c236533c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:14:02.299165 30785 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f3330e28c2d7434699b78070dd92ed73 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:02.298847 30793 catalog_manager.cc:5484] T f3330e28c2d7434699b78070dd92ed73 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 7 to 8, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:02.300254 3025 ts_tablet_manager.cc:1398] T f3e80559017d4a6ba16d34e0b7aab085 P 4665a3377574417ba0d3165c236533c0: Time spent starting tablet: real 0.069s user 0.005s sys 0.000s
I20240503 14:14:02.301210 394 raft_consensus.cc:1261] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:02.301532 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session 4665a3377574417ba0d3165c236533c0-f3e80559017d4a6ba16d34e0b7aab085 received from {username='slave'} at 127.22.242.71:41641
I20240503 14:14:02.301677 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session 4665a3377574417ba0d3165c236533c0-f3e80559017d4a6ba16d34e0b7aab085 on tablet f3e80559017d4a6ba16d34e0b7aab085 with peer 4665a3377574417ba0d3165c236533c0
W20240503 14:14:02.304133 31138 consensus_peers.cc:485] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3330e28c2d7434699b78070dd92ed73. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:02.304348 3128 consensus_queue.cc:1035] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:02.305848 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet f3330e28c2d7434699b78070dd92ed73 with delete_type TABLET_DATA_TOMBSTONED (TS 6dc4948ed7c740ab89b9e349d151b0e5 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:37226
I20240503 14:14:02.306285 394 raft_consensus.cc:2932] T f3330e28c2d7434699b78070dd92ed73 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:02.316572 30938 consensus_queue.cc:237] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:14:02.323776 3003 ts_tablet_manager.cc:902] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:14:02.327260 31072 consensus_queue.cc:237] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:14:02.337815 3056 tablet_bootstrap.cc:492] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:02.339447 3056 tablet_bootstrap.cc:492] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap complete.
I20240503 14:14:02.339788 3056 ts_tablet_manager.cc:1367] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent bootstrapping tablet: real 0.693s user 0.055s sys 0.007s
I20240503 14:14:02.340190 3153 tablet_replica.cc:330] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5: stopping tablet replica
I20240503 14:14:02.340468 3153 raft_consensus.cc:2229] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:02.340883 3153 raft_consensus.cc:2258] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:02.341977 3056 raft_consensus.cc:348] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:14:02.342342 3056 raft_consensus.cc:729] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa71cbdf0e544b81bd4e7e27196d2c91, State: Initialized, Role: LEARNER
I20240503 14:14:02.342579 3056 consensus_queue.cc:260] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:14:02.343242 3056 ts_tablet_manager.cc:1398] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:02.346429 658 raft_consensus.cc:1203] T f3e80559017d4a6ba16d34e0b7aab085 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:02.349858 3153 ts_tablet_manager.cc:1875] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:02.353322 30961 tablet_copy_service.cc:342] P 0dc902d91ccc435a82a5ede4a7729811: Request end of tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-f6aeaa3fe1cd4127a6144d85b0434057 received from {username='slave'} at 127.22.242.70:54795
I20240503 14:14:02.353461 30961 tablet_copy_service.cc:433] P 0dc902d91ccc435a82a5ede4a7729811: ending tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-f6aeaa3fe1cd4127a6144d85b0434057 on tablet f6aeaa3fe1cd4127a6144d85b0434057 with peer fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:14:02.366241 30877 consensus_peers.cc:485] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 -> Peer 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189): Couldn't send request to peer 6967bdda50404d68a5ac043b709440e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a9b84d307b8432bbdf6b70a11bf66b9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:02.367631 526 raft_consensus.cc:1261] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:02.368367 31200 raft_consensus.cc:1261] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:02.368440 2992 consensus_queue.cc:1035] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:14:02.369552 31005 consensus_peers.cc:485] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829): Couldn't send request to peer fa71cbdf0e544b81bd4e7e27196d2c91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bd38c5cccf34dc3b8e6ec5425cb5e87. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:02.369781 3092 consensus_queue.cc:1035] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:02.379964 30936 raft_consensus.cc:1261] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:02.380803 3092 consensus_queue.cc:1035] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:02.381352 31199 raft_consensus.cc:1261] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:02.382084 2992 consensus_queue.cc:1035] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:02.399811 3092 raft_consensus.cc:2932] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:14:02.405841 31202 raft_consensus.cc:2932] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:14:02.420732 30938 raft_consensus.cc:2932] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:14:02.418438 30800 catalog_manager.cc:5484] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index -1 to 4, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:14:02.440722 2992 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:14:02.457497 526 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:14:02.461687 791 raft_consensus.cc:1203] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 14:14:02.464840 30791 catalog_manager.cc:5484] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 reported cstate change: config changed from index 6 to 7, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "0dc902d91ccc435a82a5ede4a7729811" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:14:02.467242 31199 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:14:02.472514 2796 tablet_bootstrap.cc:492] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:02.473001 2796 tablet_bootstrap.cc:492] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap complete.
I20240503 14:14:02.473294 2796 ts_tablet_manager.cc:1367] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5: Time spent bootstrapping tablet: real 1.069s user 0.052s sys 0.013s
I20240503 14:14:02.488750 2796 raft_consensus.cc:348] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:02.489246 2796 raft_consensus.cc:729] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85f69eefe4294197923dff5efadcbbd5, State: Initialized, Role: LEARNER
I20240503 14:14:02.489538 2796 consensus_queue.cc:260] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:02.503034 3111 tablet_bootstrap.cc:492] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:02.503615 3111 tablet_bootstrap.cc:492] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1: Bootstrap complete.
I20240503 14:14:02.503922 3111 ts_tablet_manager.cc:1367] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1: Time spent bootstrapping tablet: real 0.225s user 0.048s sys 0.024s
I20240503 14:14:02.506455 3153 ts_tablet_manager.cc:1888] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 14:14:02.508687 3153 log.cc:1196] T f3330e28c2d7434699b78070dd92ed73 P 6dc4948ed7c740ab89b9e349d151b0e5: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/f3330e28c2d7434699b78070dd92ed73
I20240503 14:14:02.509423 3111 raft_consensus.cc:348] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:14:02.509615 30787 catalog_manager.cc:4830] TS 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221): tablet f3330e28c2d7434699b78070dd92ed73 (table rebalance_test_table_3 [id=ca6b9f7a0ded4c3aa1ea18aa2055c41b]) successfully deleted
I20240503 14:14:02.509795 3111 raft_consensus.cc:729] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6967bdda50404d68a5ac043b709440e1, State: Initialized, Role: LEARNER
I20240503 14:14:02.510133 2796 ts_tablet_manager.cc:1398] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5: Time spent starting tablet: real 0.037s user 0.003s sys 0.001s
I20240503 14:14:02.510041 3111 consensus_queue.cc:260] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:14:02.510768 3111 ts_tablet_manager.cc:1398] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:02.511077 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session 85f69eefe4294197923dff5efadcbbd5-d6b9a10c474f46bda0310ce661022b42 received from {username='slave'} at 127.22.242.72:54703
I20240503 14:14:02.511266 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session 85f69eefe4294197923dff5efadcbbd5-d6b9a10c474f46bda0310ce661022b42 on tablet d6b9a10c474f46bda0310ce661022b42 with peer 85f69eefe4294197923dff5efadcbbd5
I20240503 14:14:02.512223 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session 6967bdda50404d68a5ac043b709440e1-584769402a7e4e8a985a8f2363e2b8b9 received from {username='slave'} at 127.22.242.69:44021
I20240503 14:14:02.512398 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session 6967bdda50404d68a5ac043b709440e1-584769402a7e4e8a985a8f2363e2b8b9 on tablet 584769402a7e4e8a985a8f2363e2b8b9 with peer 6967bdda50404d68a5ac043b709440e1
I20240503 14:14:02.525877 3003 tablet_copy_client.cc:308] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:14:02.530164 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet c86ee72320b647eb84e376f8995b55d2 from peer 85f69eefe4294197923dff5efadcbbd5 ({username='slave'} at 127.22.242.72:54703)
I20240503 14:14:02.530390 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet c86ee72320b647eb84e376f8995b55d2 from peer 85f69eefe4294197923dff5efadcbbd5 at {username='slave'} at 127.22.242.72:54703: session id = 85f69eefe4294197923dff5efadcbbd5-c86ee72320b647eb84e376f8995b55d2
I20240503 14:14:02.531005 3025 ts_tablet_manager.cc:902] T e7b5dd5eb46a45989bf9ffe3144bc189 P 4665a3377574417ba0d3165c236533c0: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:14:02.534420 31092 tablet_copy_source_session.cc:205] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:02.536047 3003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86ee72320b647eb84e376f8995b55d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:02.588244 31326 consensus_queue.cc:237] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
W20240503 14:14:02.591358 31268 consensus_peers.cc:485] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be -> Peer 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189): Couldn't send request to peer 6967bdda50404d68a5ac043b709440e1. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:02.592082 658 raft_consensus.cc:1261] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:02.592684 3033 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:02.595284 30938 raft_consensus.cc:1261] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:02.596060 3033 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:02.599298 3042 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:14:02.599896 658 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:14:02.611835 30798 catalog_manager.cc:5484] T bc77a7da906b419085258f72667ff185 P 4665a3377574417ba0d3165c236533c0 reported cstate change: config changed from index 8 to 9, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:14:02.617795 526 raft_consensus.cc:1203] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:02.629021 30936 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } } }
I20240503 14:14:02.661849 3003 tablet_copy_client.cc:783] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:02.662199 3003 tablet_copy_client.cc:647] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:02.665717 3003 tablet_copy_client.cc:523] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:02.666496 3025 tablet_copy_client.cc:308] T e7b5dd5eb46a45989bf9ffe3144bc189 P 4665a3377574417ba0d3165c236533c0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:14:02.667284 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet e7b5dd5eb46a45989bf9ffe3144bc189 from peer 4665a3377574417ba0d3165c236533c0 ({username='slave'} at 127.22.242.71:41641)
I20240503 14:14:02.667488 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet e7b5dd5eb46a45989bf9ffe3144bc189 from peer 4665a3377574417ba0d3165c236533c0 at {username='slave'} at 127.22.242.71:41641: session id = 4665a3377574417ba0d3165c236533c0-e7b5dd5eb46a45989bf9ffe3144bc189
I20240503 14:14:02.670799 31092 tablet_copy_source_session.cc:205] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:02.672180 3025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b5dd5eb46a45989bf9ffe3144bc189. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:02.676606 3003 tablet_bootstrap.cc:492] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap starting.
I20240503 14:14:02.682934 3069 tablet_bootstrap.cc:492] T 91a7b8a53140429b956178d43478a541 P 4665a3377574417ba0d3165c236533c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:02.683589 3069 tablet_bootstrap.cc:492] T 91a7b8a53140429b956178d43478a541 P 4665a3377574417ba0d3165c236533c0: Bootstrap complete.
I20240503 14:14:02.688688 3025 tablet_copy_client.cc:783] T e7b5dd5eb46a45989bf9ffe3144bc189 P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:02.721117 3025 tablet_copy_client.cc:647] T e7b5dd5eb46a45989bf9ffe3144bc189 P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:02.723044 3069 ts_tablet_manager.cc:1367] T 91a7b8a53140429b956178d43478a541 P 4665a3377574417ba0d3165c236533c0: Time spent bootstrapping tablet: real 0.784s user 0.053s sys 0.014s
I20240503 14:14:02.723335 3025 tablet_copy_client.cc:523] T e7b5dd5eb46a45989bf9ffe3144bc189 P 4665a3377574417ba0d3165c236533c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:02.724423 3069 raft_consensus.cc:348] T 91a7b8a53140429b956178d43478a541 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:14:02.725116 3069 raft_consensus.cc:729] T 91a7b8a53140429b956178d43478a541 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4665a3377574417ba0d3165c236533c0, State: Initialized, Role: LEARNER
I20240503 14:14:02.725484 3069 consensus_queue.cc:260] T 91a7b8a53140429b956178d43478a541 P 4665a3377574417ba0d3165c236533c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:14:02.727408 3069 ts_tablet_manager.cc:1398] T 91a7b8a53140429b956178d43478a541 P 4665a3377574417ba0d3165c236533c0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:14:02.727644 3025 tablet_bootstrap.cc:492] T e7b5dd5eb46a45989bf9ffe3144bc189 P 4665a3377574417ba0d3165c236533c0: Bootstrap starting.
I20240503 14:14:02.786674 31221 tablet_copy_service.cc:342] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Request end of tablet copy session 4665a3377574417ba0d3165c236533c0-91a7b8a53140429b956178d43478a541 received from {username='slave'} at 127.22.242.71:44441
I20240503 14:14:02.786832 31221 tablet_copy_service.cc:433] P e7badbe63a8a4d9ca0c78818c6f4c8ce: ending tablet copy session 4665a3377574417ba0d3165c236533c0-91a7b8a53140429b956178d43478a541 on tablet 91a7b8a53140429b956178d43478a541 with peer 4665a3377574417ba0d3165c236533c0
I20240503 14:14:02.790181 3111 ts_tablet_manager.cc:902] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1: Initiating tablet copy from peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
I20240503 14:14:02.800096 3069 ts_tablet_manager.cc:902] T f3330e28c2d7434699b78070dd92ed73 P 4665a3377574417ba0d3165c236533c0: Initiating tablet copy from peer e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
I20240503 14:14:02.807188 31326 consensus_queue.cc:237] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:02.853242 3111 tablet_copy_client.cc:308] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:46013
I20240503 14:14:02.857116 526 raft_consensus.cc:1261] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:02.857198 30961 tablet_copy_service.cc:140] P 0dc902d91ccc435a82a5ede4a7729811: Received BeginTabletCopySession request for tablet 1a9b84d307b8432bbdf6b70a11bf66b9 from peer 6967bdda50404d68a5ac043b709440e1 ({username='slave'} at 127.22.242.69:60815)
I20240503 14:14:02.857383 30961 tablet_copy_service.cc:161] P 0dc902d91ccc435a82a5ede4a7729811: Beginning new tablet copy session on tablet 1a9b84d307b8432bbdf6b70a11bf66b9 from peer 6967bdda50404d68a5ac043b709440e1 at {username='slave'} at 127.22.242.69:60815: session id = 6967bdda50404d68a5ac043b709440e1-1a9b84d307b8432bbdf6b70a11bf66b9
I20240503 14:14:02.857748 3033 consensus_queue.cc:1035] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:14:02.858953 31268 consensus_peers.cc:485] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23476c4bcf6e4b4085d59d29739b6edb. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:02.860752 31199 raft_consensus.cc:1261] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:02.863219 3033 consensus_queue.cc:1035] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:02.867559 30961 tablet_copy_source_session.cc:205] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:02.868144 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:02.869103 3111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a9b84d307b8432bbdf6b70a11bf66b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:02.879241 3003 tablet_bootstrap.cc:492] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:02.881933 3003 tablet_bootstrap.cc:492] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap complete.
I20240503 14:14:02.883966 3003 ts_tablet_manager.cc:1367] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5: Time spent bootstrapping tablet: real 0.208s user 0.073s sys 0.020s
I20240503 14:14:02.885160 3003 raft_consensus.cc:348] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:02.885007 525 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:02.885424 31326 consensus_queue.cc:237] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:14:02.889453 3003 raft_consensus.cc:729] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85f69eefe4294197923dff5efadcbbd5, State: Initialized, Role: LEARNER
I20240503 14:14:02.889964 3003 consensus_queue.cc:260] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:02.890933 3003 ts_tablet_manager.cc:1398] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:14:02.892143 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session 85f69eefe4294197923dff5efadcbbd5-c86ee72320b647eb84e376f8995b55d2 received from {username='slave'} at 127.22.242.72:54703
I20240503 14:14:02.892360 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session 85f69eefe4294197923dff5efadcbbd5-c86ee72320b647eb84e376f8995b55d2 on tablet c86ee72320b647eb84e376f8995b55d2 with peer 85f69eefe4294197923dff5efadcbbd5
I20240503 14:14:02.893255 3111 tablet_copy_client.cc:783] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:02.893527 3111 tablet_copy_client.cc:647] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:02.896538 394 raft_consensus.cc:1203] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:02.898007 3111 tablet_copy_client.cc:523] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:02.898546 791 raft_consensus.cc:1203] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:02.902225 3111 tablet_bootstrap.cc:492] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1: Bootstrap starting.
I20240503 14:14:02.883777 3033 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:02.908914 658 raft_consensus.cc:1203] T 91a7b8a53140429b956178d43478a541 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:02.912027 30800 catalog_manager.cc:5484] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91 reported cstate change: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:02.914254 31200 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:02.915571 3236 ts_tablet_manager.cc:902] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:14:02.956349 3092 raft_consensus.cc:1050] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER 4665a3377574417ba0d3165c236533c0 to VOTER
I20240503 14:14:02.957242 3092 consensus_queue.cc:237] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:14:02.961892 31330 raft_consensus.cc:1261] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:02.962100 791 raft_consensus.cc:1261] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:02.962675 3092 consensus_queue.cc:1035] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:02.962898 658 raft_consensus.cc:1261] T f3e80559017d4a6ba16d34e0b7aab085 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:02.963451 3098 consensus_queue.cc:1035] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:02.966428 3092 consensus_queue.cc:1035] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:02.970130 3069 tablet_copy_client.cc:308] T f3330e28c2d7434699b78070dd92ed73 P 4665a3377574417ba0d3165c236533c0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44297
I20240503 14:14:02.970754 790 raft_consensus.cc:1261] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:02.971370 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:02.971760 3042 consensus_queue.cc:1035] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:02.971941 31221 tablet_copy_service.cc:140] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Received BeginTabletCopySession request for tablet f3330e28c2d7434699b78070dd92ed73 from peer 4665a3377574417ba0d3165c236533c0 ({username='slave'} at 127.22.242.71:44441)
I20240503 14:14:02.972084 31221 tablet_copy_service.cc:161] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Beginning new tablet copy session on tablet f3330e28c2d7434699b78070dd92ed73 from peer 4665a3377574417ba0d3165c236533c0 at {username='slave'} at 127.22.242.71:44441: session id = 4665a3377574417ba0d3165c236533c0-f3330e28c2d7434699b78070dd92ed73
I20240503 14:14:02.975610 31221 tablet_copy_source_session.cc:205] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 14:14:02.976912 31268 consensus_peers.cc:485] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69b741aad35d4bfc8fea5b8e21d4a733. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:02.980468 31199 raft_consensus.cc:1261] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:02.981380 3042 consensus_queue.cc:1035] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:02.987877 3042 raft_consensus.cc:2932] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:02.993355 31200 raft_consensus.cc:2932] T 69b741aad35d4bfc8fea5b8e21d4a733 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:03.001945 3236 tablet_copy_client.cc:308] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:14:03.002756 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87 from peer fa71cbdf0e544b81bd4e7e27196d2c91 ({username='slave'} at 127.22.242.70:40181)
I20240503 14:14:03.002939 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87 from peer fa71cbdf0e544b81bd4e7e27196d2c91 at {username='slave'} at 127.22.242.70:40181: session id = fa71cbdf0e544b81bd4e7e27196d2c91-3bd38c5cccf34dc3b8e6ec5425cb5e87
I20240503 14:14:03.003160 3069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3330e28c2d7434699b78070dd92ed73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:03.006274 31092 tablet_copy_source_session.cc:205] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:03.009860 30800 catalog_manager.cc:5484] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be reported cstate change: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:03.011746 640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:03.016863 3099 raft_consensus.cc:1050] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER 85f69eefe4294197923dff5efadcbbd5 to VOTER
I20240503 14:14:03.017537 3236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:03.017843 3099 consensus_queue.cc:237] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:03.083123 394 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:14:03.083876 3092 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:03.084347 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:03.085213 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:14:03.098395 792 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:14:03.099537 3091 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:03.101194 3098 raft_consensus.cc:2932] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:14:03.103101 31326 raft_consensus.cc:2932] T f3e80559017d4a6ba16d34e0b7aab085 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:14:03.108608 3069 tablet_copy_client.cc:783] T f3330e28c2d7434699b78070dd92ed73 P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:03.110211 658 raft_consensus.cc:2932] T f3e80559017d4a6ba16d34e0b7aab085 P 4665a3377574417ba0d3165c236533c0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:14:03.111174 30800 catalog_manager.cc:5484] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 7 to 8, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:03.112808 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:03.119397 31330 raft_consensus.cc:1261] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:14:03.120075 3099 consensus_queue.cc:1035] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:03.121587 3236 tablet_copy_client.cc:783] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:03.126356 3236 tablet_copy_client.cc:647] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:03.128867 3236 tablet_copy_client.cc:523] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:03.135880 3069 tablet_copy_client.cc:647] T f3330e28c2d7434699b78070dd92ed73 P 4665a3377574417ba0d3165c236533c0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:03.143319 3069 tablet_copy_client.cc:523] T f3330e28c2d7434699b78070dd92ed73 P 4665a3377574417ba0d3165c236533c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:03.150043 507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:03.172701 3236 tablet_bootstrap.cc:492] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap starting.
I20240503 14:14:03.176542 3025 tablet_bootstrap.cc:492] T e7b5dd5eb46a45989bf9ffe3144bc189 P 4665a3377574417ba0d3165c236533c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:03.177680 3025 tablet_bootstrap.cc:492] T e7b5dd5eb46a45989bf9ffe3144bc189 P 4665a3377574417ba0d3165c236533c0: Bootstrap complete.
I20240503 14:14:03.178826 3069 tablet_bootstrap.cc:492] T f3330e28c2d7434699b78070dd92ed73 P 4665a3377574417ba0d3165c236533c0: Bootstrap starting.
I20240503 14:14:03.178972 3025 ts_tablet_manager.cc:1367] T e7b5dd5eb46a45989bf9ffe3144bc189 P 4665a3377574417ba0d3165c236533c0: Time spent bootstrapping tablet: real 0.450s user 0.059s sys 0.005s
I20240503 14:14:03.180860 3025 raft_consensus.cc:348] T e7b5dd5eb46a45989bf9ffe3144bc189 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:14:03.181149 3025 raft_consensus.cc:729] T e7b5dd5eb46a45989bf9ffe3144bc189 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4665a3377574417ba0d3165c236533c0, State: Initialized, Role: LEARNER
I20240503 14:14:03.182313 3025 consensus_queue.cc:260] T e7b5dd5eb46a45989bf9ffe3144bc189 P 4665a3377574417ba0d3165c236533c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
W20240503 14:14:03.183161 3255 log.cc:924] Time spent T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20240503 14:14:03.187938 787 raft_consensus.cc:2932] T 69b741aad35d4bfc8fea5b8e21d4a733 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4665a3377574417ba0d3165c236533c0 (127.22.242.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } } }
I20240503 14:14:03.190608 3025 ts_tablet_manager.cc:1398] T e7b5dd5eb46a45989bf9ffe3144bc189 P 4665a3377574417ba0d3165c236533c0: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20240503 14:14:03.244557 3280 raft_consensus.cc:1050] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811: attempting to promote NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 to VOTER
I20240503 14:14:03.245870 3280 consensus_queue.cc:237] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:14:03.247248 3273 ts_tablet_manager.cc:902] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: Initiating tablet copy from peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:14:03.254112 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session 4665a3377574417ba0d3165c236533c0-e7b5dd5eb46a45989bf9ffe3144bc189 received from {username='slave'} at 127.22.242.71:41641
I20240503 14:14:03.254295 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session 4665a3377574417ba0d3165c236533c0-e7b5dd5eb46a45989bf9ffe3144bc189 on tablet e7b5dd5eb46a45989bf9ffe3144bc189 with peer 4665a3377574417ba0d3165c236533c0
I20240503 14:14:03.255745 526 raft_consensus.cc:1261] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:03.256963 3122 consensus_queue.cc:1035] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:03.265297 3003 ts_tablet_manager.cc:902] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5: Initiating tablet copy from peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
I20240503 14:14:03.265587 393 raft_consensus.cc:1261] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:03.267426 3099 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:03.271950 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:03.276454 3281 consensus_queue.cc:1035] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.010s
I20240503 14:14:03.276516 393 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:03.286899 31072 raft_consensus.cc:1261] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:03.293926 3281 consensus_queue.cc:1035] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:03.308913 787 raft_consensus.cc:1203] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:03.311070 3003 tablet_copy_client.cc:308] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.68:37063
I20240503 14:14:03.311923 31352 tablet_copy_service.cc:140] P 82cfdc130877418dbe9d980d368973be: Received BeginTabletCopySession request for tablet 23476c4bcf6e4b4085d59d29739b6edb from peer 85f69eefe4294197923dff5efadcbbd5 ({username='slave'} at 127.22.242.72:36333)
I20240503 14:14:03.312263 31352 tablet_copy_service.cc:161] P 82cfdc130877418dbe9d980d368973be: Beginning new tablet copy session on tablet 23476c4bcf6e4b4085d59d29739b6edb from peer 85f69eefe4294197923dff5efadcbbd5 at {username='slave'} at 127.22.242.72:36333: session id = 85f69eefe4294197923dff5efadcbbd5-23476c4bcf6e4b4085d59d29739b6edb
I20240503 14:14:03.316224 31352 tablet_copy_source_session.cc:205] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:03.317548 791 raft_consensus.cc:2932] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:14:03.318050 395 raft_consensus.cc:2932] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:14:03.323901 3122 raft_consensus.cc:2932] T f6aeaa3fe1cd4127a6144d85b0434057 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:14:03.325603 31326 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
W20240503 14:14:03.327378 31005 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.65:46013, user_credentials={real_user=slave}} blocked reactor thread for 66348.5us
I20240503 14:14:03.328685 787 raft_consensus.cc:2932] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:03.329499 3003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23476c4bcf6e4b4085d59d29739b6edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:03.329830 30793 catalog_manager.cc:5484] T 63b2120788a54af78f02449f46bf23c7 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 9 to 10, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:03.334602 526 raft_consensus.cc:2932] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:14:03.337502 30800 catalog_manager.cc:5484] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 reported cstate change: config changed from index 7 to 8, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0dc902d91ccc435a82a5ede4a7729811" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
I20240503 14:14:03.344509 31072 raft_consensus.cc:2932] T f6aeaa3fe1cd4127a6144d85b0434057 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } }
W20240503 14:14:03.355350 31006 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.67:44297, user_credentials={real_user=slave}} blocked reactor thread for 53333.7us
I20240503 14:14:03.385357 3273 tablet_copy_client.cc:308] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.68:37063
I20240503 14:14:03.387760 3092 raft_consensus.cc:1050] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER 6967bdda50404d68a5ac043b709440e1 to VOTER
I20240503 14:14:03.388747 3092 consensus_queue.cc:237] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:14:03.390527 3236 tablet_bootstrap.cc:492] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:03.392385 3236 tablet_bootstrap.cc:492] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap complete.
I20240503 14:14:03.392762 3236 ts_tablet_manager.cc:1367] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent bootstrapping tablet: real 0.220s user 0.057s sys 0.006s
I20240503 14:14:03.395013 3003 tablet_copy_client.cc:783] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:03.394999 3236 raft_consensus.cc:348] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:14:03.400873 3236 raft_consensus.cc:729] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa71cbdf0e544b81bd4e7e27196d2c91, State: Initialized, Role: LEARNER
I20240503 14:14:03.400911 31326 raft_consensus.cc:1261] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:03.401127 3236 consensus_queue.cc:260] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:14:03.401620 3099 consensus_queue.cc:1035] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:03.402686 3236 ts_tablet_manager.cc:1398] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240503 14:14:03.404299 3003 tablet_copy_client.cc:647] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:03.407745 394 raft_consensus.cc:1261] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:03.408380 3099 consensus_queue.cc:1035] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:03.409068 3003 tablet_copy_client.cc:523] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:03.417933 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-3bd38c5cccf34dc3b8e6ec5425cb5e87 received from {username='slave'} at 127.22.242.70:40181
I20240503 14:14:03.418094 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-3bd38c5cccf34dc3b8e6ec5425cb5e87 on tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87 with peer fa71cbdf0e544b81bd4e7e27196d2c91
I20240503 14:14:03.418735 3003 tablet_bootstrap.cc:492] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5: Bootstrap starting.
I20240503 14:14:03.420020 526 raft_consensus.cc:1261] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:03.420779 3290 consensus_queue.cc:1035] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:03.421883 3098 raft_consensus.cc:1050] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER 85f69eefe4294197923dff5efadcbbd5 to VOTER
I20240503 14:14:03.422868 3098 consensus_queue.cc:237] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:03.425779 3025 ts_tablet_manager.cc:902] T 69b741aad35d4bfc8fea5b8e21d4a733 P 4665a3377574417ba0d3165c236533c0: Initiating tablet copy from peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
F20240503 14:14:03.426146 3025 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0f45824b1fff4302abd03c1fb48c6c7b.queue_time_us, tablet-download-pool-0f45824b1fff4302abd03c1fb48c6c7b.queue_time_us) (tablet-download-pool-0f45824b1fff4302abd03c1fb48c6c7b.run_cpu_time_us, tablet-download-pool-0f45824b1fff4302abd03c1fb48c6c7b.run_cpu_time_us) (tablet-download-pool-0f45824b1fff4302abd03c1fb48c6c7b.run_wall_time_us, tablet-download-pool-0f45824b1fff4302abd03c1fb48c6c7b.run_wall_time_us) (tablet-download-pool-4ff2fa69f79644c5b93beeba32fc9ca0.queue_time_us, tablet-download-pool-4ff2fa69f79644c5b93beeba32fc9ca0.queue_time_us) (tablet-download-pool-4ff2fa69f79644c5b93beeba32fc9ca0.run_cpu_time_us, tablet-download-pool-4ff2fa69f79644c5b93beeba32fc9ca0.run_cpu_time_us) (tablet-download-pool-4ff2fa69f79644c5b93beeba32fc9ca0.run_wall_time_us, tablet-download-pool-4ff2fa69f79644c5b93beeba32fc9ca0.run_wall_time_us) (tablet-download-pool-5051c4e81ea14146a9f1d82b7f5b5f1d.queue_time_us, tablet-download-pool-5051c4e81ea14146a9f1d82b7f5b5f1d.queue_time_us) (tablet-download-pool-5051c4e81ea14146a9f1d82b7f5b5f1d.run_cpu_time_us, tablet-download-pool-5051c4e81ea14146a9f1d82b7f5b5f1d.run_cpu_time_us) (tablet-download-pool-5051c4e81ea14146a9f1d82b7f5b5f1d.run_wall_time_us, tablet-download-pool-5051c4e81ea14146a9f1d82b7f5b5f1d.run_wall_time_us) (tablet-download-pool-55cf34de74694f47ad3f9f4d9bdffae4.queue_time_us, tablet-download-pool-55cf34de74694f47ad3f9f4d9bdffae4.queue_time_us) (tablet-download-pool-55cf34de74694f47ad3f9f4d9bdffae4.run_cpu_time_us, tablet-download-pool-55cf34de74694f47ad3f9f4d9bdffae4.run_cpu_time_us) (tablet-download-pool-55cf34de74694f47ad3f9f4d9bdffae4.run_wall_time_us, tablet-download-pool-55cf34de74694f47ad3f9f4d9bdffae4.run_wall_time_us) (tablet-download-pool-5abcdc783c6144f28b9f8a9e1e162006.queue_time_us, tablet-download-pool-5abcdc783c6144f28b9f8a9e1e162006.queue_time_us) (tablet-download-pool-5abcdc783c6144f28b9f8a9e1e162006.run_cpu_time_us, tablet-download-pool-5abcdc783c6144f28b9f8a9e1e162006.run_cpu_time_us) (tablet-download-pool-5abcdc783c6144f28b9f8a9e1e162006.run_wall_time_us, tablet-download-pool-5abcdc783c6144f28b9f8a9e1e162006.run_wall_time_us) (tablet-download-pool-6234489254fc4882a39b7df28ddf673f.queue_time_us, tablet-download-pool-6234489254fc4882a39b7df28ddf673f.queue_time_us) (tablet-download-pool-6234489254fc4882a39b7df28ddf673f.run_cpu_time_us, tablet-download-pool-6234489254fc4882a39b7df28ddf673f.run_cpu_time_us) (tablet-download-pool-6234489254fc4882a39b7df28ddf673f.run_wall_time_us, tablet-download-pool-6234489254fc4882a39b7df28ddf673f.run_wall_time_us) (tablet-download-pool-626fcb185af0446b89ae82066e08981d.queue_time_us, tablet-download-pool-626fcb185af0446b89ae82066e08981d.queue_time_us) (tablet-download-pool-626fcb185af0446b89ae82066e08981d.run_cpu_time_us, tablet-download-pool-626fcb185af0446b89ae82066e08981d.run_cpu_time_us) (tablet-download-pool-626fcb185af0446b89ae82066e08981d.run_wall_time_us, tablet-download-pool-626fcb185af0446b89ae82066e08981d.run_wall_time_us) (tablet-download-pool-69b741aad35d4bfc8fea5b8e21d4a733.queue_time_us, tablet-download-pool-69b741aad35d4bfc8fea5b8e21d4a733.queue_time_us) (tablet-download-pool-91a7b8a53140429b956178d43478a541.queue_time_us, tablet-download-pool-91a7b8a53140429b956178d43478a541.queue_time_us) (tablet-download-pool-91a7b8a53140429b956178d43478a541.run_cpu_time_us, tablet-download-pool-91a7b8a53140429b956178d43478a541.run_cpu_time_us) (tablet-download-pool-91a7b8a53140429b956178d43478a541.run_wall_time_us, tablet-download-pool-91a7b8a53140429b956178d43478a541.run_wall_time_us) (tablet-download-pool-bc77a7da906b419085258f72667ff185.queue_time_us, tablet-download-pool-bc77a7da906b419085258f72667ff185.queue_time_us) (tablet-download-pool-bc77a7da906b419085258f72667ff185.run_cpu_time_us, tablet-download-pool-bc77a7da906b419085258f72667ff185.run_cpu_time_us) (tablet-download-pool-bc77a7da906b419085258f72667ff185.run_wall_time_us, tablet-download-pool-bc77a7da906b419085258f72667ff185.run_wall_time_us) (tablet-download-pool-da9dd5cd81114817b0f898c6b8fb3c07.queue_time_us, tablet-download-pool-da9dd5cd81114817b0f898c6b8fb3c07.queue_time_us) (tablet-download-pool-da9dd5cd81114817b0f898c6b8fb3c07.run_cpu_time_us, tablet-download-pool-da9dd5cd81114817b0f898c6b8fb3c07.run_cpu_time_us) (tablet-download-pool-da9dd5cd81114817b0f898c6b8fb3c07.run_wall_time_us, tablet-download-pool-da9dd5cd81114817b0f898c6b8fb3c07.run_wall_time_us) (tablet-download-pool-e7b5dd5eb46a45989bf9ffe3144bc189.queue_time_us, tablet-download-pool-e7b5dd5eb46a45989bf9ffe3144bc189.queue_time_us) (tablet-download-pool-e7b5dd5eb46a45989bf9ffe3144bc189.run_cpu_time_us, tablet-download-pool-e7b5dd5eb46a45989bf9ffe3144bc189.run_cpu_time_us) (tablet-download-pool-e7b5dd5eb46a45989bf9ffe3144bc189.run_wall_time_us, tablet-download-pool-e7b5dd5eb46a45989bf9ffe3144bc189.run_wall_time_us) (tablet-download-pool-ebd6dc710e144e61a5fae3911f25ee5c.queue_time_us, tablet-download-pool-ebd6dc710e144e61a5fae3911f25ee5c.queue_time_us) (tablet-download-pool-ebd6dc710e144e61a5fae3911f25ee5c.run_cpu_time_us, tablet-download-pool-ebd6dc710e144e61a5fae3911f25ee5c.run_cpu_time_us) (tablet-download-pool-ebd6dc710e144e61a5fae3911f25ee5c.run_wall_time_us, tablet-download-pool-ebd6dc710e144e61a5fae3911f25ee5c.run_wall_time_us) (tablet-download-pool-f3330e28c2d7434699b78070dd92ed73.queue_time_us, tablet-download-pool-f3330e28c2d7434699b78070dd92ed73.queue_time_us) (tablet-download-pool-f3330e28c2d7434699b78070dd92ed73.run_cpu_time_us, tablet-download-pool-f3330e28c2d7434699b78070dd92ed73.run_cpu_time_us) (tablet-download-pool-f3330e28c2d7434699b78070dd92ed73.run_wall_time_us, tablet-download-pool-f3330e28c2d7434699b78070dd92ed73.run_wall_time_us) (tablet-download-pool-f3e80559017d4a6ba16d34e0b7aab085.queue_time_us, tablet-download-pool-f3e80559017d4a6ba16d34e0b7aab085.queue_time_us) (tablet-download-pool-f3e80559017d4a6ba16d34e0b7aab085.run_cpu_time_us, tablet-download-pool-f3e80559017d4a6ba16d34e0b7aab085.run_cpu_time_us) (tablet-download-pool-f3e80559017d4a6ba16d34e0b7aab085.run_wall_time_us, tablet-download-pool-f3e80559017d4a6ba16d34e0b7aab085.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f9ad5ef0dcd google::LogMessage::Fail() at ??:0
@ 0x7f9ad5ef4b93 google::LogMessage::SendToLog() at ??:0
I20240503 14:14:03.434582 31352 tablet_copy_service.cc:140] P 82cfdc130877418dbe9d980d368973be: Received BeginTabletCopySession request for tablet bc77a7da906b419085258f72667ff185 from peer 6967bdda50404d68a5ac043b709440e1 ({username='slave'} at 127.22.242.69:57313)
I20240503 14:14:03.434787 31352 tablet_copy_service.cc:161] P 82cfdc130877418dbe9d980d368973be: Beginning new tablet copy session on tablet bc77a7da906b419085258f72667ff185 from peer 6967bdda50404d68a5ac043b709440e1 at {username='slave'} at 127.22.242.69:57313: session id = 6967bdda50404d68a5ac043b709440e1-bc77a7da906b419085258f72667ff185
I20240503 14:14:03.438148 31352 tablet_copy_source_session.cc:205] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7f9ad5ef07cc google::LogMessage::Flush() at ??:0
I20240503 14:14:03.440574 3273 ts_tablet_manager.cc:1875] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:14:03.440547 31072 consensus_queue.cc:237] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
@ 0x7f9ad5ef1f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 14:14:03.443137 790 raft_consensus.cc:1261] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:03.443388 30936 raft_consensus.cc:1261] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:03.443989 31200 raft_consensus.cc:1261] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:03.443974 3296 consensus_queue.cc:1035] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:03.445070 3289 consensus_queue.cc:1035] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:03.445608 3289 consensus_queue.cc:1035] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
@ 0x7f9ad8c31742 kudu::TraceMetrics::InternName() at ??:0
I20240503 14:14:03.458604 3092 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:03.463532 31326 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
@ 0x7f9ad8bfdf8b kudu::ThreadPool::ThreadPool() at ??:0
I20240503 14:14:03.471947 395 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
@ 0x7f9ad8bf7cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 14:14:03.493386 3128 raft_consensus.cc:1050] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce: attempting to promote NON_VOTER 4665a3377574417ba0d3165c236533c0 to VOTER
I20240503 14:14:03.494366 3128 consensus_queue.cc:237] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
@ 0x7f9ae804b8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 14:14:03.499300 526 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:03.504335 30796 catalog_manager.cc:5484] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 7 to 8, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
@ 0x7f9ae8063daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 14:14:03.514604 31071 consensus_queue.cc:237] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:14:03.520789 3069 tablet_bootstrap.cc:492] T f3330e28c2d7434699b78070dd92ed73 P 4665a3377574417ba0d3165c236533c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:03.521462 3069 tablet_bootstrap.cc:492] T f3330e28c2d7434699b78070dd92ed73 P 4665a3377574417ba0d3165c236533c0: Bootstrap complete.
I20240503 14:14:03.522429 526 raft_consensus.cc:1261] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:03.523195 3092 consensus_queue.cc:1035] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
@ 0x7f9ae818e480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 14:14:03.524329 3069 ts_tablet_manager.cc:1367] T f3330e28c2d7434699b78070dd92ed73 P 4665a3377574417ba0d3165c236533c0: Time spent bootstrapping tablet: real 0.346s user 0.059s sys 0.011s
I20240503 14:14:03.526144 3286 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:03.526880 525 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:03.536926 3069 raft_consensus.cc:348] T f3330e28c2d7434699b78070dd92ed73 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:14:03.537324 3069 raft_consensus.cc:729] T f3330e28c2d7434699b78070dd92ed73 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4665a3377574417ba0d3165c236533c0, State: Initialized, Role: LEARNER
I20240503 14:14:03.537636 3069 consensus_queue.cc:260] T f3330e28c2d7434699b78070dd92ed73 P 4665a3377574417ba0d3165c236533c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: true } }
I20240503 14:14:03.538848 395 raft_consensus.cc:1261] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:03.539497 3099 consensus_queue.cc:1035] T 584769402a7e4e8a985a8f2363e2b8b9 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:03.539690 3069 ts_tablet_manager.cc:1398] T f3330e28c2d7434699b78070dd92ed73 P 4665a3377574417ba0d3165c236533c0: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20240503 14:14:03.540989 31221 tablet_copy_service.cc:342] P e7badbe63a8a4d9ca0c78818c6f4c8ce: Request end of tablet copy session 4665a3377574417ba0d3165c236533c0-f3330e28c2d7434699b78070dd92ed73 received from {username='slave'} at 127.22.242.71:44441
I20240503 14:14:03.541195 31221 tablet_copy_service.cc:433] P e7badbe63a8a4d9ca0c78818c6f4c8ce: ending tablet copy session 4665a3377574417ba0d3165c236533c0-f3330e28c2d7434699b78070dd92ed73 on tablet f3330e28c2d7434699b78070dd92ed73 with peer 4665a3377574417ba0d3165c236533c0
I20240503 14:14:03.541379 393 raft_consensus.cc:2932] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
@ 0x7f9ae81c6a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 14:14:03.546654 3092 raft_consensus.cc:2932] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:03.547422 31199 raft_consensus.cc:2932] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:03.554704 30800 catalog_manager.cc:5484] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:03.554698 30796 catalog_manager.cc:5484] T 584769402a7e4e8a985a8f2363e2b8b9 P 6967bdda50404d68a5ac043b709440e1 reported cstate change: config changed from index 8 to 9, VOTER 82cfdc130877418dbe9d980d368973be (127.22.242.68) evicted. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:03.553977 30936 raft_consensus.cc:2932] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
@ 0x7f9ae81a9981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240503 14:14:03.569792 30936 raft_consensus.cc:1261] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:03.570415 3099 consensus_queue.cc:1035] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:03.570716 31200 raft_consensus.cc:1261] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
@ 0x7f9ae81a970e std::_Function_handler<>::_M_invoke() at ??:0
I20240503 14:14:03.571249 3296 consensus_queue.cc:1035] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240503 14:14:03.579738 526 raft_consensus.cc:1261] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
@ 0x7f9ad8c01b75 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 14:14:03.580509 3296 consensus_queue.cc:1035] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:03.581909 31311 tablet_service.cc:1514] Processing DeleteTablet for tablet 584769402a7e4e8a985a8f2363e2b8b9 with delete_type TABLET_DATA_TOMBSTONED (TS 82cfdc130877418dbe9d980d368973be not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48102
@ 0x7f9ad8c0519c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240503 14:14:03.586364 3273 ts_tablet_manager.cc:1888] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240503 14:14:03.588809 3273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc77a7da906b419085258f72667ff185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:03.591491 3099 raft_consensus.cc:2932] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
@ 0x7f9ad8c05019 std::_Function_handler<>::_M_invoke() at ??:0
I20240503 14:14:03.592730 658 raft_consensus.cc:1261] T 91a7b8a53140429b956178d43478a541 P 4665a3377574417ba0d3165c236533c0 [term 1 LEARNER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:03.593506 2090 consensus_queue.cc:1035] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240503 14:14:03.601065 30938 raft_consensus.cc:2932] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:03.603498 30787 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 584769402a7e4e8a985a8f2363e2b8b9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
@ 0x7f9ad8be001e kudu::Thread::SuperviseThread() at ??:0
I20240503 14:14:03.607434 527 raft_consensus.cc:1261] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
@ 0x7f9adbf656db start_thread at ??:0
I20240503 14:14:03.609864 3273 tablet_copy_client.cc:783] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:03.610159 3273 tablet_copy_client.cc:647] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:03.610858 2090 consensus_queue.cc:1035] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
@ 0x7f9ad4da371f clone at ??:0
I20240503 14:14:03.613061 3273 tablet_copy_client.cc:523] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:03.613937 30936 raft_consensus.cc:1261] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer e7badbe63a8a4d9ca0c78818c6f4c8ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:03.614842 2090 consensus_queue.cc:1035] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:14:03.624819 31008 consensus_peers.cc:485] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bd38c5cccf34dc3b8e6ec5425cb5e87. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:03.627213 3337 tablet_replica.cc:330] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be: stopping tablet replica
I20240503 14:14:03.627414 3289 raft_consensus.cc:1050] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 to VOTER
I20240503 14:14:03.627419 3337 raft_consensus.cc:2229] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:03.628401 3289 consensus_queue.cc:237] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:03.627821 3337 raft_consensus.cc:2258] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:03.631341 31200 raft_consensus.cc:1261] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:03.631796 3092 consensus_queue.cc:1035] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:03.631815 3337 ts_tablet_manager.cc:1875] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:03.631011 525 raft_consensus.cc:2932] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
W20240503 14:14:03.633142 31008 consensus_peers.cc:485] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bd38c5cccf34dc3b8e6ec5425cb5e87. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:03.645417 30793 catalog_manager.cc:5484] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 4 to 5, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added, now has a pending config: VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67), VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66), VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65), VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70), NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72). New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:03.657773 526 raft_consensus.cc:1261] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:03.657685 3002 raft_consensus.cc:2932] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:14:03.660827 3098 consensus_queue.cc:1035] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:03.665980 30940 raft_consensus.cc:2932] T 91a7b8a53140429b956178d43478a541 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:14:03.668323 790 raft_consensus.cc:2932] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:03.668481 30794 catalog_manager.cc:5484] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 7 to 8, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:03.671742 527 raft_consensus.cc:2932] T 91a7b8a53140429b956178d43478a541 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4665a3377574417ba0d3165c236533c0 (127.22.242.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:14:03.681718 31199 raft_consensus.cc:2932] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:03.679571 3337 ts_tablet_manager.cc:1888] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:03.682786 3337 log.cc:1196] T 584769402a7e4e8a985a8f2363e2b8b9 P 82cfdc130877418dbe9d980d368973be: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/wals/584769402a7e4e8a985a8f2363e2b8b9
I20240503 14:14:03.683851 30786 catalog_manager.cc:4830] TS 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063): tablet 584769402a7e4e8a985a8f2363e2b8b9 (table rebalance_test_table_8 [id=c243ced42cda4e3996901964941b50f4]) successfully deleted
I20240503 14:14:03.695288 30936 raft_consensus.cc:1261] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:03.696877 3092 consensus_queue.cc:1035] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:03.702545 3092 raft_consensus.cc:2932] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:03.710150 30938 raft_consensus.cc:2932] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:03.715387 31072 consensus_queue.cc:237] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:03.718780 527 raft_consensus.cc:2932] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:03.717089 30792 catalog_manager.cc:5484] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 5 to 6, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67), VOTER 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66), VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65), VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70), NON_VOTER 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72). New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:03.726979 31200 raft_consensus.cc:1261] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:03.727643 3092 consensus_queue.cc:1035] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:03.730947 3098 raft_consensus.cc:2932] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:03.738267 30787 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d6b9a10c474f46bda0310ce661022b42 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:03.737788 31200 raft_consensus.cc:2932] T d6b9a10c474f46bda0310ce661022b42 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:03.739893 30794 catalog_manager.cc:5484] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:03.744673 790 raft_consensus.cc:1261] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:03.746308 3273 tablet_bootstrap.cc:492] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: Bootstrap starting.
I20240503 14:14:03.746282 30920 tablet_service.cc:1514] Processing DeleteTablet for tablet d6b9a10c474f46bda0310ce661022b42 with delete_type TABLET_DATA_TOMBSTONED (TS 0dc902d91ccc435a82a5ede4a7729811 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56642
I20240503 14:14:03.750615 3092 consensus_queue.cc:1035] T d6b9a10c474f46bda0310ce661022b42 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 14:14:03.752759 790 raft_consensus.cc:2932] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:03.757838 3351 tablet_replica.cc:330] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811: stopping tablet replica
I20240503 14:14:03.758095 3351 raft_consensus.cc:2229] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:03.758412 3351 raft_consensus.cc:2258] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:03.766449 3351 ts_tablet_manager.cc:1875] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:03.795557 31072 consensus_queue.cc:237] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:03.799186 527 raft_consensus.cc:1261] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:03.799865 3098 consensus_queue.cc:1035] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:14:03.804868 31008 consensus_peers.cc:485] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72:36545): Couldn't send request to peer 85f69eefe4294197923dff5efadcbbd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bd38c5cccf34dc3b8e6ec5425cb5e87. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:03.805495 31200 raft_consensus.cc:2932] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:03.813441 31199 raft_consensus.cc:1261] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:03.814234 3098 consensus_queue.cc:1035] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:03.824044 3098 raft_consensus.cc:2932] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:03.828826 31199 raft_consensus.cc:2932] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:03.830500 30787 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:04.133667 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:03.832405 30794 catalog_manager.cc:5484] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 6 to 7, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:03.832973 525 raft_consensus.cc:2932] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } } }
I20240503 14:14:04.240922 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:04.321734 507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:14:04.347815 31268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.72:36545, user_credentials={real_user=slave}} blocked reactor thread for 47235.4us
I20240503 14:14:03.932081 3111 tablet_bootstrap.cc:492] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:04.095571 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:03.856758 3351 ts_tablet_manager.cc:1888] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:04.112524 3273 tablet_bootstrap.cc:492] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:04.367079 3351 log.cc:1196] T d6b9a10c474f46bda0310ce661022b42 P 0dc902d91ccc435a82a5ede4a7729811: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/d6b9a10c474f46bda0310ce661022b42
I20240503 14:14:04.367317 3111 tablet_bootstrap.cc:492] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1: Bootstrap complete.
I20240503 14:14:04.367466 3273 tablet_bootstrap.cc:492] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: Bootstrap complete.
I20240503 14:14:04.367638 3111 ts_tablet_manager.cc:1367] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1: Time spent bootstrapping tablet: real 1.466s user 0.068s sys 0.056s
I20240503 14:14:04.367751 3273 ts_tablet_manager.cc:1367] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: Time spent bootstrapping tablet: real 0.622s user 0.059s sys 0.004s
I20240503 14:14:04.034276 30920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:04.211604 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87 with delete_type TABLET_DATA_TOMBSTONED (TS 0dc902d91ccc435a82a5ede4a7729811 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56642
I20240503 14:14:04.368366 30784 catalog_manager.cc:4830] TS 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): tablet d6b9a10c474f46bda0310ce661022b42 (table rebalance_test_table_5 [id=04f8dfb749c04486895d6b0f7c206446]) successfully deleted
I20240503 14:14:04.368647 3111 raft_consensus.cc:348] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:14:04.368896 3111 raft_consensus.cc:729] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6967bdda50404d68a5ac043b709440e1, State: Initialized, Role: LEARNER
I20240503 14:14:04.374328 3351 tablet_replica.cc:330] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811: stopping tablet replica
I20240503 14:14:04.374567 3351 raft_consensus.cc:2229] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:04.374940 3351 raft_consensus.cc:2258] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:04.379936 3351 ts_tablet_manager.cc:1875] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:04.387269 3351 ts_tablet_manager.cc:1888] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:14:03.836884 3092 raft_consensus.cc:1050] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER 85f69eefe4294197923dff5efadcbbd5 to VOTER
I20240503 14:14:04.118698 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:14:04.441462 3351 log.cc:1196] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 0dc902d91ccc435a82a5ede4a7729811: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/3bd38c5cccf34dc3b8e6ec5425cb5e87
I20240503 14:14:04.442365 3092 consensus_queue.cc:237] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:04.442795 30784 catalog_manager.cc:4830] TS 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87 (table rebalance_test_table_7 [id=5b67f0a6a8d34286994597065f21a92b]) successfully deleted
I20240503 14:14:04.180035 31072 consensus_queue.cc:237] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:14:04.328728 3286 raft_consensus.cc:1050] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER 85f69eefe4294197923dff5efadcbbd5 to VOTER
I20240503 14:14:04.449410 31199 raft_consensus.cc:1261] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:04.496711 31329 raft_consensus.cc:1261] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 14:14:04.532209 31138 connection.cc:535] client connection to 127.22.242.71:46453 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:04.532557 31138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:04.774892 507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:04.896623 508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:14:04.448519 3286 raft_consensus.cc:1054] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5: Unable to promote non-voter 85f69eefe4294197923dff5efadcbbd5: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:04.913681 506 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:04.369158 3111 consensus_queue.cc:260] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:14:04.497357 30938 raft_consensus.cc:1261] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:04.806573 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:04.450919 3296 consensus_queue.cc:1035] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:04.371136 3273 raft_consensus.cc:348] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
W20240503 14:14:04.533509 30877 connection.cc:535] client connection to 127.22.242.71:46453 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:04.880623 31124 kernel_stack_watchdog.cc:198] Thread 31138 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 170ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa9f1200980 (unknown) at ??:0
@ 0x7fa9ea022255 __write at ??:0
@ 0x7fa9e9f9d28d _IO_file_write at ??:0
@ 0x7fa9e9f9dbff _IO_file_xsputn at ??:0
@ 0x7fa9e9f91a57 fwrite at ??:0
@ 0x8793e7 __interceptor_fwrite.part.56 at ??:?
@ 0x7fa9eb1848a0 google::LogMessage::SendToLog() at ??:0
@ 0x7fa9eb1807cc google::LogMessage::Flush() at ??:0
@ 0x7fa9eb180a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fa9f4ac0bf4 kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7fa9f4ac04ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fa9f4aac4e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa9f4aad365 kudu::rpc::OutboundCall::SetFailed() at ??:0
@ 0x7fa9f4a42bb3 kudu::rpc::Connection::Shutdown() at ??:0
@ 0x7fa9f4acae51 kudu::rpc::ReactorThread::DestroyConnection() at ??:0
W20240503 14:14:04.923373 30877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:04.497437 3455 consensus_queue.cc:1035] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:03.991693 3003 tablet_bootstrap.cc:492] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:04.256968 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:14:04.533109 31008 connection.cc:535] client connection to 127.22.242.71:46453 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:04.764326 31051 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 14:14:04.872822 30994 kernel_stack_watchdog.cc:198] Thread 3454 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/consensus/log.cc:407 for 192ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fb33eace980 (unknown) at ??:0
@ 0x7fb33a74ca61 sys_futex() at ??:0
@ 0x7fb33a74c8df base::internal::SpinLockDelay() at ??:0
@ 0x7fb33a74c3ae base::SpinLock::SlowLock() at ??:0
@ 0x955226 base::SpinLock::Lock() at ??:?
@ 0x9551ee kudu::simple_spinlock::lock() at ??:?
@ 0x955114 std::lock_guard<>::lock_guard() at ??:?
@ 0x7fb346ea6f9d kudu::consensus::PeerMessageQueue::ResponseFromPeer() at ??:0
@ 0x7fb346ea67a4 kudu::consensus::PeerMessageQueue::LocalPeerAppendFinished() at ??:0
@ 0x7fb346ebe4da kudu::consensus::PeerMessageQueue::AppendOperations()::$_1::operator()() at ??:0
@ 0x7fb346ebe242 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fb34ec6b17e std::function<>::operator()() at ??:0
@ 0x7fb346f0b57e kudu::consensus::LogCache::LogCallback() at ??:0
@ 0x7fb346f1035f kudu::consensus::LogCache::AppendOperations()::$_0::operator()() at ??:0
@ 0x7fb346f100e2 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fb34ec6b17e std::function<>::operator()() at ??:0
I20240503 14:14:04.873394 31050 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 14:14:04.924324 31008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:04.294628 3425 ts_tablet_manager.cc:902] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:14:04.447574 395 raft_consensus.cc:1261] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:04.451474 790 raft_consensus.cc:1261] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 14:14:04.532902 31268 connection.cc:535] client connection to 127.22.242.71:46453 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:04.926200 3003 tablet_bootstrap.cc:492] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5: Bootstrap complete.
W20240503 14:14:04.926147 31268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:04.864352 31311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:04.926519 3003 ts_tablet_manager.cc:1367] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5: Time spent bootstrapping tablet: real 1.508s user 0.054s sys 0.015s
I20240503 14:14:04.901799 31312 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:04.927415 3003 raft_consensus.cc:348] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:04.927683 3003 raft_consensus.cc:729] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85f69eefe4294197923dff5efadcbbd5, State: Initialized, Role: LEARNER
I20240503 14:14:04.927876 3003 consensus_queue.cc:260] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:04.930209 3455 consensus_queue.cc:1035] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:04.940079 30921 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:04.943804 30921 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:04.948359 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:04.951186 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:04.959829 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:14:04.965073 31138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.71:46453, user_credentials={real_user=slave}} blocked reactor thread for 432576us
W20240503 14:14:04.981768 31138 consensus_peers.cc:485] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:04.989815 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:05.013885 3273 raft_consensus.cc:729] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6967bdda50404d68a5ac043b709440e1, State: Initialized, Role: LEARNER
I20240503 14:14:05.014220 3273 consensus_queue.cc:260] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:14:05.022611 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:05.034984 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:05.039286 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:14:05.076862 31268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.71:46453, user_credentials={real_user=slave}} blocked reactor thread for 150728us
W20240503 14:14:05.077980 30877 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.71:46453, user_credentials={real_user=slave}} blocked reactor thread for 154696us
W20240503 14:14:05.084363 31138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.69:42189, user_credentials={real_user=slave}} blocked reactor thread for 54373.8us
I20240503 14:14:05.086311 3455 raft_consensus.cc:2932] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:14:05.091485 30936 raft_consensus.cc:2932] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:14:05.109499 31330 raft_consensus.cc:2932] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } }
I20240503 14:14:05.120033 3111 ts_tablet_manager.cc:1398] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1: Time spent starting tablet: real 0.752s user 0.005s sys 0.014s
W20240503 14:14:05.120298 3111 ts_tablet_manager.cc:1430] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1: Tablet startup took 752ms
W20240503 14:14:05.120384 3111 ts_tablet_manager.cc:1432] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1: Trace:
0503 14:14:02.789841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:14:02.789911 (+ 70us) service_pool.cc:225] Handling call
0503 14:14:02.790263 (+ 352us) ts_tablet_manager.cc:903] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1: Initiating tablet copy from peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013)
0503 14:14:02.868530 (+ 78267us) tablet_copy_client.cc:353] Tablet copy session begun
0503 14:14:02.872936 (+ 4406us) tablet_metadata.cc:641] Metadata flushed
0503 14:14:02.873038 (+ 102us) tablet_copy_client.cc:482] Wrote new tablet metadata
0503 14:14:02.892808 (+ 19770us) tablet_copy_client.cc:403] Wrote new consensus metadata
0503 14:14:02.892933 (+ 125us) tablet_replica.cc:205] Creating consensus instance
0503 14:14:02.893063 (+ 130us) ts_tablet_manager.cc:958] Replica registered.
0503 14:14:02.893087 (+ 24us) inbound_call.cc:173] Queueing success response
0503 14:14:02.902068 (+ 8981us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0503 14:14:04.367716 (+1465648us) ts_tablet_manager.cc:1399] Starting tablet replica
0503 14:14:04.367941 (+ 225us) tablet_replica.cc:259] Starting instrumentation
0503 14:14:04.368213 (+ 272us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":201,"mutex_wait_us":3,"raft.queue_time_us":160419,"raft.run_cpu_time_us":59,"raft.run_wall_time_us":60,"spinlock_wait_cycles":1280,"tablet-copy.queue_time_us":45,"tablet-download-pool-1a9b84d307b8432bbdf6b70a11bf66b9.queue_time_us":2606,"tablet-download-pool-1a9b84d307b8432bbdf6b70a11bf66b9.run_cpu_time_us":506,"tablet-download-pool-1a9b84d307b8432bbdf6b70a11bf66b9.run_wall_time_us":1591,"thread_start_us":259957,"threads_started":5}
I20240503 14:14:05.154093 3003 ts_tablet_manager.cc:1398] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5: Time spent starting tablet: real 0.227s user 0.022s sys 0.045s
I20240503 14:14:05.154731 31352 tablet_copy_service.cc:342] P 82cfdc130877418dbe9d980d368973be: Request end of tablet copy session 85f69eefe4294197923dff5efadcbbd5-23476c4bcf6e4b4085d59d29739b6edb received from {username='slave'} at 127.22.242.72:36333
I20240503 14:14:05.154876 31352 tablet_copy_service.cc:433] P 82cfdc130877418dbe9d980d368973be: ending tablet copy session 85f69eefe4294197923dff5efadcbbd5-23476c4bcf6e4b4085d59d29739b6edb on tablet 23476c4bcf6e4b4085d59d29739b6edb with peer 85f69eefe4294197923dff5efadcbbd5
I20240503 14:14:05.156605 3273 ts_tablet_manager.cc:1398] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: Time spent starting tablet: real 0.789s user 0.002s sys 0.014s
W20240503 14:14:05.157742 3273 ts_tablet_manager.cc:1430] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: Tablet startup took 789ms
W20240503 14:14:05.158126 3273 ts_tablet_manager.cc:1432] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: Trace:
0503 14:14:03.124324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:14:03.124380 (+ 56us) service_pool.cc:225] Handling call
0503 14:14:03.247391 (+123011us) ts_tablet_manager.cc:903] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1: Initiating tablet copy from peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063)
0503 14:14:03.440109 (+192718us) tablet_copy_client.cc:353] Tablet copy session begun
0503 14:14:03.509729 (+ 69620us) tablet_metadata.cc:641] Metadata flushed
0503 14:14:03.586168 (+ 76439us) tablet_metadata.cc:641] Metadata flushed
0503 14:14:03.586599 (+ 431us) tablet_copy_client.cc:448] Replaced tombstoned tablet metadata.
0503 14:14:03.603197 (+ 16598us) tablet_copy_client.cc:403] Wrote new consensus metadata
0503 14:14:03.603338 (+ 141us) tablet_replica.cc:205] Creating consensus instance
0503 14:14:03.603498 (+ 160us) ts_tablet_manager.cc:958] Replica registered.
0503 14:14:03.603525 (+ 27us) inbound_call.cc:173] Queueing success response
0503 14:14:03.745373 (+141848us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0503 14:14:04.367849 (+622476us) ts_tablet_manager.cc:1399] Starting tablet replica
0503 14:14:04.368100 (+ 251us) tablet_replica.cc:259] Starting instrumentation
0503 14:14:04.368263 (+ 163us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":90,"mutex_wait_us":2504,"raft.queue_time_us":64672,"raft.run_cpu_time_us":14,"raft.run_wall_time_us":15,"tablet-copy.queue_time_us":122232,"tablet-download-pool-bc77a7da906b419085258f72667ff185.queue_time_us":43,"tablet-download-pool-bc77a7da906b419085258f72667ff185.run_cpu_time_us":481,"tablet-download-pool-bc77a7da906b419085258f72667ff185.run_wall_time_us":2126,"thread_start_us":401452,"threads_started":6}
I20240503 14:14:05.159653 31352 tablet_copy_service.cc:342] P 82cfdc130877418dbe9d980d368973be: Request end of tablet copy session 6967bdda50404d68a5ac043b709440e1-bc77a7da906b419085258f72667ff185 received from {username='slave'} at 127.22.242.69:57313
I20240503 14:14:05.159807 31352 tablet_copy_service.cc:433] P 82cfdc130877418dbe9d980d368973be: ending tablet copy session 6967bdda50404d68a5ac043b709440e1-bc77a7da906b419085258f72667ff185 on tablet bc77a7da906b419085258f72667ff185 with peer 6967bdda50404d68a5ac043b709440e1
W20240503 14:14:05.163125 31008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.71:46453, user_credentials={real_user=slave}} blocked reactor thread for 238868us
I20240503 14:14:05.167577 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:05.179982 30794 catalog_manager.cc:5484] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:05.205086 3425 tablet_copy_client.cc:308] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:14:05.205860 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87 from peer 85f69eefe4294197923dff5efadcbbd5 ({username='slave'} at 127.22.242.72:54703)
I20240503 14:14:05.206044 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87 from peer 85f69eefe4294197923dff5efadcbbd5 at {username='slave'} at 127.22.242.72:54703: session id = 85f69eefe4294197923dff5efadcbbd5-3bd38c5cccf34dc3b8e6ec5425cb5e87
W20240503 14:14:05.206101 30877 consensus_peers.cc:485] T 5051c4e81ea14146a9f1d82b7f5b5f1d P 0dc902d91ccc435a82a5ede4a7729811 -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:05.209092 30921 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:05.209273 31092 tablet_copy_source_session.cc:205] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:05.210657 3425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:14:05.210798 31268 consensus_peers.cc:462] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111)
W20240503 14:14:05.210947 31268 consensus_peers.cc:485] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:05.211052 31268 consensus_peers.cc:485] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:05.211140 31268 consensus_peers.cc:485] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:05.211228 31268 consensus_peers.cc:485] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:05.214790 31268 consensus_peers.cc:485] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:05.223071 393 raft_consensus.cc:1203] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240503 14:14:05.224448 31266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.67:44297, user_credentials={real_user=slave}} blocked reactor thread for 58131.7us
I20240503 14:14:05.228646 30961 tablet_copy_service.cc:342] P 0dc902d91ccc435a82a5ede4a7729811: Request end of tablet copy session 6967bdda50404d68a5ac043b709440e1-1a9b84d307b8432bbdf6b70a11bf66b9 received from {username='slave'} at 127.22.242.69:60815
I20240503 14:14:05.228791 30961 tablet_copy_service.cc:433] P 0dc902d91ccc435a82a5ede4a7729811: ending tablet copy session 6967bdda50404d68a5ac043b709440e1-1a9b84d307b8432bbdf6b70a11bf66b9 on tablet 1a9b84d307b8432bbdf6b70a11bf66b9 with peer 6967bdda50404d68a5ac043b709440e1
I20240503 14:14:05.232146 3455 consensus_queue.cc:1035] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:05.237892 3455 consensus_queue.cc:1035] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:14:05.239234 31008 consensus_peers.cc:485] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:05.247283 394 raft_consensus.cc:1203] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240503 14:14:05.259737 31008 consensus_peers.cc:485] T f3e80559017d4a6ba16d34e0b7aab085 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:05.259910 31008 consensus_peers.cc:485] T ebd6dc710e144e61a5fae3911f25ee5c P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:05.260003 31008 consensus_peers.cc:485] T 5abcdc783c6144f28b9f8a9e1e162006 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:05.260102 31008 consensus_peers.cc:485] T 4ff2fa69f79644c5b93beeba32fc9ca0 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:05.277019 31050 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:14:05.281674 3296 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:05.287482 31199 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:05.294498 394 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:05.330367 31071 consensus_queue.cc:237] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
W20240503 14:14:05.334501 31005 consensus_peers.cc:485] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829): Couldn't send request to peer fa71cbdf0e544b81bd4e7e27196d2c91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da9dd5cd81114817b0f898c6b8fb3c07. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:05.334981 31330 raft_consensus.cc:1261] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:05.335290 30936 raft_consensus.cc:1261] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:05.335587 3296 consensus_queue.cc:1035] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:05.337239 3296 consensus_queue.cc:1035] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:05.341535 3098 raft_consensus.cc:2932] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:14:05.342018 30936 raft_consensus.cc:2932] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:14:05.347182 30787 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet da9dd5cd81114817b0f898c6b8fb3c07 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:14:05.348532 31330 raft_consensus.cc:2932] T da9dd5cd81114817b0f898c6b8fb3c07 P 82cfdc130877418dbe9d980d368973be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
W20240503 14:14:05.350260 31008 consensus_peers.cc:485] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:05.364024 30800 catalog_manager.cc:5484] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce reported cstate change: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:05.364306 30794 catalog_manager.cc:5484] T da9dd5cd81114817b0f898c6b8fb3c07 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 6 to 7, NON_VOTER fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } } }
I20240503 14:14:05.402593 31312 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:05.434597 790 raft_consensus.cc:1203] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:05.444463 3425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.22.242.66:54333 (request call id 102) took 1148 ms. Trace:
I20240503 14:14:05.446404 3425 rpcz_store.cc:276] 0503 14:14:04.291839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:14:04.291893 (+ 54us) service_pool.cc:225] Handling call
0503 14:14:04.924700 (+632807us) ts_tablet_manager.cc:903] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
0503 14:14:05.210123 (+285423us) tablet_copy_client.cc:353] Tablet copy session begun
0503 14:14:05.214117 (+ 3994us) tablet_metadata.cc:641] Metadata flushed
0503 14:14:05.214262 (+ 145us) tablet_copy_client.cc:482] Wrote new tablet metadata
0503 14:14:05.433541 (+219279us) tablet_copy_client.cc:403] Wrote new consensus metadata
0503 14:14:05.433673 (+ 132us) tablet_replica.cc:205] Creating consensus instance
0503 14:14:05.440362 (+ 6689us) ts_tablet_manager.cc:958] Replica registered.
0503 14:14:05.440382 (+ 20us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":106,"tablet-copy.queue_time_us":2452,"thread_start_us":282268,"threads_started":5}
I20240503 14:14:05.447535 3425 tablet_copy_client.cc:783] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:05.447811 3425 tablet_copy_client.cc:647] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:05.459069 3425 tablet_copy_client.cc:523] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:05.462749 3425 tablet_bootstrap.cc:492] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap starting.
I20240503 14:14:05.483191 31071 consensus_queue.cc:237] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:05.485775 791 raft_consensus.cc:1261] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:05.487613 394 raft_consensus.cc:1261] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:05.488266 3455 consensus_queue.cc:1035] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:05.491358 3455 consensus_queue.cc:1035] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:05.497120 791 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:05.504619 3098 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:05.518498 394 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:05.525241 30796 catalog_manager.cc:5484] T c86ee72320b647eb84e376f8995b55d2 P 6967bdda50404d68a5ac043b709440e1 reported cstate change: config changed from index 8 to 9, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:05.529922 31180 tablet_service.cc:1514] Processing DeleteTablet for tablet c86ee72320b647eb84e376f8995b55d2 with delete_type TABLET_DATA_TOMBSTONED (TS e7badbe63a8a4d9ca0c78818c6f4c8ce not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45854
I20240503 14:14:05.529359 791 raft_consensus.cc:2932] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:05.550253 3580 tablet_replica.cc:330] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce: stopping tablet replica
I20240503 14:14:05.550477 3580 raft_consensus.cc:2229] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:05.550755 3580 raft_consensus.cc:2258] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:05.556700 3580 ts_tablet_manager.cc:1875] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:05.577067 30787 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c86ee72320b647eb84e376f8995b55d2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 14:14:05.589582 31268 consensus_peers.cc:462] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111)
I20240503 14:14:05.621104 3554 raft_consensus.cc:1050] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be: attempting to promote NON_VOTER 6967bdda50404d68a5ac043b709440e1 to VOTER
I20240503 14:14:05.622018 3554 consensus_queue.cc:237] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:14:05.630955 3425 tablet_bootstrap.cc:492] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:05.631418 3425 tablet_bootstrap.cc:492] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5: Bootstrap complete.
I20240503 14:14:05.631671 3425 ts_tablet_manager.cc:1367] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5: Time spent bootstrapping tablet: real 0.169s user 0.059s sys 0.018s
I20240503 14:14:05.632599 3425 raft_consensus.cc:348] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:05.632866 3425 raft_consensus.cc:729] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85f69eefe4294197923dff5efadcbbd5, State: Initialized, Role: LEARNER
I20240503 14:14:05.633082 3425 consensus_queue.cc:260] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: NON_VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: true } }
I20240503 14:14:05.633605 3425 ts_tablet_manager.cc:1398] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:14:05.638027 394 raft_consensus.cc:1261] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:14:05.638536 30936 raft_consensus.cc:1261] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:14:05.641191 3042 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240503 14:14:05.641796 3554 consensus_queue.cc:1035] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:05.645985 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session 85f69eefe4294197923dff5efadcbbd5-3bd38c5cccf34dc3b8e6ec5425cb5e87 received from {username='slave'} at 127.22.242.72:54703
I20240503 14:14:05.646203 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session 85f69eefe4294197923dff5efadcbbd5-3bd38c5cccf34dc3b8e6ec5425cb5e87 on tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87 with peer 85f69eefe4294197923dff5efadcbbd5
W20240503 14:14:05.649900 31268 consensus_peers.cc:485] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:05.661573 3281 raft_consensus.cc:1050] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811: attempting to promote NON_VOTER 6967bdda50404d68a5ac043b709440e1 to VOTER
I20240503 14:14:05.662832 3281 consensus_queue.cc:237] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:14:05.668422 3554 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:05.674616 394 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:05.676123 30938 raft_consensus.cc:2932] T bc77a7da906b419085258f72667ff185 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:05.679850 31199 raft_consensus.cc:1261] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:05.682240 3542 consensus_queue.cc:1035] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:05.683494 30796 catalog_manager.cc:5484] T bc77a7da906b419085258f72667ff185 P 6967bdda50404d68a5ac043b709440e1 reported cstate change: config changed from index 9 to 10, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:05.685266 525 raft_consensus.cc:1261] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:05.919894 841 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 14:13:46.928181 847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:46.928573 847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:46.941766 847 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:50.293443 847 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 14:13:51.929466 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:51.937382 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca6b9f7a0ded4c3aa1ea18aa2055c41b: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:51.937538 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:51.938025 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6d48b36155b41d3a75fe51d7d204ea7: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:51.938159 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:51.938341 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:51.938463 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table acc4ce6b628f4f498f830c4572e30d65: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:51.938638 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:51.938743 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5b67f0a6a8d34286994597065f21a92b: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:51.938877 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:51.939010 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:51.939134 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6d48b36155b41d3a75fe51d7d204ea7: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:51.939226 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:51.939338 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 6967bdda50404d68a5ac043b709440e1
I20240503 14:13:52.122853 847 rebalancer_tool.cc:1074] tablet 2f82073122a74268a464387659dd90bf: '0dc902d91ccc435a82a5ede4a7729811' -> '6967bdda50404d68a5ac043b709440e1' move scheduled
I20240503 14:13:52.299836 847 rebalancer_tool.cc:1074] tablet c837def6d76242f08b20a448541e64d6: '82cfdc130877418dbe9d980d368973be' -> 'fa71cbdf0e544b81bd4e7e27196d2c91' move scheduled
I20240503 14:13:52.427425 847 rebalancer_tool.cc:1074] tablet 1bb75d7d9499475ca60c633c0801d639: '6dc4948ed7c740ab89b9e349d151b0e5' -> '85f69eefe4294197923dff5efadcbbd5' move scheduled
I20240503 14:13:52.573801 847 rebalancer_tool.cc:1074] tablet 0f45824b1fff4302abd03c1fb48c6c7b: 'e7badbe63a8a4d9ca0c78818c6f4c8ce' -> '4665a3377574417ba0d3165c236533c0' move scheduled
I20240503 14:13:52.742686 847 rebalancer_tool.cc:1074] tablet f17feb9b64214eaebcbe31dc45c75756: '0dc902d91ccc435a82a5ede4a7729811' -> '6967bdda50404d68a5ac043b709440e1' move scheduled
I20240503 14:13:52.999936 847 rebalancer_tool.cc:1074] tablet 91a7b8a53140429b956178d43478a541: '82cfdc130877418dbe9d980d368973be' -> 'fa71cbdf0e544b81bd4e7e27196d2c91' move scheduled
I20240503 14:13:53.422027 847 rebalancer_tool.cc:1074] tablet 69b741aad35d4bfc8fea5b8e21d4a733: '6dc4948ed7c740ab89b9e349d151b0e5' -> '85f69eefe4294197923dff5efadcbbd5' move scheduled
I20240503 14:13:53.850409 847 rebalancer_tool.cc:1074] tablet bc77a7da906b419085258f72667ff185: 'e7badbe63a8a4d9ca0c78818c6f4c8ce' -> '4665a3377574417ba0d3165c236533c0' move scheduled
I20240503 14:13:53.917327 847 rebalancer_tool.cc:1074] tablet 626fcb185af0446b89ae82066e08981d: '0dc902d91ccc435a82a5ede4a7729811' -> 'fa71cbdf0e544b81bd4e7e27196d2c91' move scheduled
I20240503 14:13:54.191522 847 rebalancer_tool.cc:1074] tablet c86ee72320b647eb84e376f8995b55d2: '82cfdc130877418dbe9d980d368973be' -> '6967bdda50404d68a5ac043b709440e1' move scheduled
I20240503 14:13:54.392395 847 rebalancer_tool.cc:1074] tablet 6234489254fc4882a39b7df28ddf673f: '6dc4948ed7c740ab89b9e349d151b0e5' -> '4665a3377574417ba0d3165c236533c0' move scheduled
I20240503 14:13:54.660403 847 rebalancer_tool.cc:1074] tablet f3e80559017d4a6ba16d34e0b7aab085: 'e7badbe63a8a4d9ca0c78818c6f4c8ce' -> '85f69eefe4294197923dff5efadcbbd5' move scheduled
I20240503 14:13:55.174752 847 rebalancer_tool.cc:1074] tablet 584769402a7e4e8a985a8f2363e2b8b9: '0dc902d91ccc435a82a5ede4a7729811' -> 'fa71cbdf0e544b81bd4e7e27196d2c91' move scheduled
I20240503 14:13:55.623003 847 rebalancer_tool.cc:1074] tablet f6aeaa3fe1cd4127a6144d85b0434057: '82cfdc130877418dbe9d980d368973be' -> '6967bdda50404d68a5ac043b709440e1' move scheduled
I20240503 14:13:55.800930 847 rebalancer_tool.cc:1074] tablet dd5869fd8f8d41d99cf531edb7df3750: '6dc4948ed7c740ab89b9e349d151b0e5' -> 'fa71cbdf0e544b81bd4e7e27196d2c91' move scheduled
I20240503 14:13:55.898113 847 rebalancer_tool.cc:1084] tablet e7b5dd5eb46a45989bf9ffe3144bc189: 'e7badbe63a8a4d9ca0c78818c6f4c8ce' -> '4665a3377574417ba0d3165c236533c0' move ignored: Not found: Peer with uuid e7badbe63a8a4d9ca0c78818c6f4c8ce not found in consensus config
I20240503 14:13:55.898288 847 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:13:58.795717 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:58.795881 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:58.796003 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:58.796105 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca6b9f7a0ded4c3aa1ea18aa2055c41b: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:58.824761 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table acc4ce6b628f4f498f830c4572e30d65: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:58.825012 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca6b9f7a0ded4c3aa1ea18aa2055c41b: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:58.825140 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5b67f0a6a8d34286994597065f21a92b: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:58.825213 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:58.825306 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 0dc902d91ccc435a82a5ede4a7729811
W20240503 14:13:58.825423 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:58.825520 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:58.825584 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 4665a3377574417ba0d3165c236533c0
I20240503 14:13:59.205384 847 rebalancer_tool.cc:1074] tablet 43221a82a1124bf8840c6369f27f47f2: 'e7badbe63a8a4d9ca0c78818c6f4c8ce' -> '85f69eefe4294197923dff5efadcbbd5' move scheduled
I20240503 14:13:59.331318 847 rebalancer_tool.cc:1074] tablet da9dd5cd81114817b0f898c6b8fb3c07: '6dc4948ed7c740ab89b9e349d151b0e5' -> '4665a3377574417ba0d3165c236533c0' move scheduled
I20240503 14:13:59.676384 847 rebalancer_tool.cc:1084] tablet 55cf34de74694f47ad3f9f4d9bdffae4: '0dc902d91ccc435a82a5ede4a7729811' -> '85f69eefe4294197923dff5efadcbbd5' move ignored: Illegal state: tablet 55cf34de74694f47ad3f9f4d9bdffae4: replica 85f69eefe4294197923dff5efadcbbd5 is already present
I20240503 14:13:59.676505 847 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:14:00.884966 2907 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.22.242.126:34247, user_credentials={real_user=slave}} blocked reactor thread for 47087.6us
W20240503 14:14:00.954339 2907 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.22.242.126:34247, user_credentials={real_user=slave}} blocked reactor thread for 62210.9us
W20240503 14:14:02.758031 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:14:02.758275 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 0dc902d91ccc435a82a5ede4a7729811
W20240503 14:14:02.758603 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:14:02.758783 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:14:02.758989 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5b67f0a6a8d34286994597065f21a92b: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:14:02.759297 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server fa71cbdf0e544b81bd4e7e27196d2c91 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:14:02.759482 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server fa71cbdf0e544b81bd4e7e27196d2c91 to server 6dc4948ed7c740ab89b9e349d151b0e5
W20240503 14:14:02.759613 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6d48b36155b41d3a75fe51d7d204ea7: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:14:02.759778 847 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 6967bdda50404d68a5ac043b709440e1
I20240503 14:14:03.596886 847 rebalancer_tool.cc:1074] tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87: '0dc902d91ccc435a82a5ede4a7729811' -> '85f69eefe4294197923dff5efadcbbd5' move scheduled
I20240503 14:14:03.862594 847 rebalancer_tool.cc:1084] tablet 1a9b84d307b8432bbdf6b70a11bf66b9: 'e7badbe63a8a4d9ca0c78818c6f4c8ce' -> '6967bdda50404d68a5ac043b709440e1' move ignored: Illegal state: tablet 1a9b84d307b8432bbdf6b70a11bf66b9: replica 6967bdda50404d68a5ac043b709440e1 is already present
I20240503 14:14:03.862752 847 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): unacceptable health status UNAVAILABLE
I20240503 14:14:05.841923 792 raft_consensus.cc:1203] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:05.711364 3580 ts_tablet_manager.cc:1888] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:05.992769 3580 log.cc:1196] T c86ee72320b647eb84e376f8995b55d2 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/c86ee72320b647eb84e376f8995b55d2
I20240503 14:14:05.694906 394 raft_consensus.cc:1261] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1 [term 1 LEARNER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:05.685946 3542 consensus_queue.cc:1035] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:14:05.996580 31138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.242.69:42189, user_credentials={real_user=slave}} blocked reactor thread for 70252.2us
I20240503 14:14:05.997052 30785 catalog_manager.cc:4830] TS e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297): tablet c86ee72320b647eb84e376f8995b55d2 (table rebalance_test_table_7 [id=5b67f0a6a8d34286994597065f21a92b]) successfully deleted
I20240503 14:14:05.972841 3547 raft_consensus.cc:1050] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be: attempting to promote NON_VOTER 85f69eefe4294197923dff5efadcbbd5 to VOTER
I20240503 14:14:05.999480 3547 consensus_queue.cc:237] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:06.000319 3542 consensus_queue.cc:1035] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:05.956245 3626 ts_tablet_manager.cc:902] T da9dd5cd81114817b0f898c6b8fb3c07 P fa71cbdf0e544b81bd4e7e27196d2c91: Initiating tablet copy from peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
I20240503 14:14:06.028252 3542 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:06.032063 527 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:06.035764 792 raft_consensus.cc:1261] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:06.037082 3554 consensus_queue.cc:1035] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:06.038452 31199 raft_consensus.cc:1261] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:06.039698 526 raft_consensus.cc:1261] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:06.041456 3042 consensus_queue.cc:1035] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:06.042068 3042 consensus_queue.cc:1035] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:06.046334 394 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:06.057943 30796 catalog_manager.cc:5484] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1 reported cstate change: config changed from index 7 to 8, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0dc902d91ccc435a82a5ede4a7729811" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:06.061442 31200 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6967bdda50404d68a5ac043b709440e1 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:06.092943 3547 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:06.093438 30938 consensus_queue.cc:237] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:14:06.097321 791 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:06.103806 394 raft_consensus.cc:1261] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:06.104175 30796 catalog_manager.cc:5484] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5 reported cstate change: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:06.105927 31200 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:06.120537 3542 consensus_queue.cc:1035] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:06.126207 3594 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:06.133422 30784 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1a9b84d307b8432bbdf6b70a11bf66b9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:06.141968 30798 catalog_manager.cc:5484] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 reported cstate change: config changed from index 8 to 9, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "0dc902d91ccc435a82a5ede4a7729811" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:06.142441 31330 consensus_queue.cc:237] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:06.144376 792 raft_consensus.cc:1261] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:06.145236 3042 consensus_queue.cc:1035] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:06.146656 528 raft_consensus.cc:1261] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 82cfdc130877418dbe9d980d368973be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:06.147596 526 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:06.148988 3554 consensus_queue.cc:1035] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:06.150149 3042 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P 82cfdc130877418dbe9d980d368973be [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:06.151234 790 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:06.154745 31180 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a9b84d307b8432bbdf6b70a11bf66b9 with delete_type TABLET_DATA_TOMBSTONED (TS e7badbe63a8a4d9ca0c78818c6f4c8ce not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45854
I20240503 14:14:06.155139 3580 tablet_replica.cc:330] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce: stopping tablet replica
I20240503 14:14:06.155406 3580 raft_consensus.cc:2229] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:06.155586 30786 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 23476c4bcf6e4b4085d59d29739b6edb with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:06.155673 3580 raft_consensus.cc:2258] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:06.157552 3580 ts_tablet_manager.cc:1875] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:06.162870 30798 catalog_manager.cc:5484] T 23476c4bcf6e4b4085d59d29739b6edb P 85f69eefe4294197923dff5efadcbbd5 reported cstate change: config changed from index 8 to 9, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "82cfdc130877418dbe9d980d368973be" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
W20240503 14:14:06.173243 31268 consensus_peers.cc:462] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111)
I20240503 14:14:06.178781 31180 tablet_service.cc:1514] Processing DeleteTablet for tablet 23476c4bcf6e4b4085d59d29739b6edb with delete_type TABLET_DATA_TOMBSTONED (TS e7badbe63a8a4d9ca0c78818c6f4c8ce not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45854
I20240503 14:14:06.181831 3580 ts_tablet_manager.cc:1888] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:06.184796 3580 log.cc:1196] T 1a9b84d307b8432bbdf6b70a11bf66b9 P e7badbe63a8a4d9ca0c78818c6f4c8ce: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/1a9b84d307b8432bbdf6b70a11bf66b9
I20240503 14:14:06.186473 30785 catalog_manager.cc:4830] TS e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297): tablet 1a9b84d307b8432bbdf6b70a11bf66b9 (table rebalance_test_table_8 [id=c243ced42cda4e3996901964941b50f4]) successfully deleted
I20240503 14:14:06.190593 3580 tablet_replica.cc:330] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce: stopping tablet replica
I20240503 14:14:06.191335 3580 raft_consensus.cc:2229] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:06.192399 525 raft_consensus.cc:1261] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 0dc902d91ccc435a82a5ede4a7729811: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:06.193425 3594 consensus_queue.cc:1035] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 0dc902d91ccc435a82a5ede4a7729811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:06.195276 525 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:06.207099 3580 raft_consensus.cc:2258] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:06.208500 3580 ts_tablet_manager.cc:1875] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:06.220499 772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:06.225572 394 raft_consensus.cc:2932] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } }
I20240503 14:14:06.240262 3626 tablet_copy_client.cc:308] T da9dd5cd81114817b0f898c6b8fb3c07 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:33221
I20240503 14:14:06.241109 31092 tablet_copy_service.cc:140] P 6dc4948ed7c740ab89b9e349d151b0e5: Received BeginTabletCopySession request for tablet da9dd5cd81114817b0f898c6b8fb3c07 from peer fa71cbdf0e544b81bd4e7e27196d2c91 ({username='slave'} at 127.22.242.70:40181)
I20240503 14:14:06.241307 31092 tablet_copy_service.cc:161] P 6dc4948ed7c740ab89b9e349d151b0e5: Beginning new tablet copy session on tablet da9dd5cd81114817b0f898c6b8fb3c07 from peer fa71cbdf0e544b81bd4e7e27196d2c91 at {username='slave'} at 127.22.242.70:40181: session id = fa71cbdf0e544b81bd4e7e27196d2c91-da9dd5cd81114817b0f898c6b8fb3c07
I20240503 14:14:06.251506 31092 tablet_copy_source_session.cc:205] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:06.257474 528 raft_consensus.cc:2932] T 23476c4bcf6e4b4085d59d29739b6edb P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:06.261267 3626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9dd5cd81114817b0f898c6b8fb3c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:06.265656 3286 raft_consensus.cc:1050] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5: attempting to promote NON_VOTER 85f69eefe4294197923dff5efadcbbd5 to VOTER
I20240503 14:14:06.266708 3286 consensus_queue.cc:237] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:06.270922 31312 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:06.279867 3580 ts_tablet_manager.cc:1888] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:06.284795 3580 log.cc:1196] T 23476c4bcf6e4b4085d59d29739b6edb P e7badbe63a8a4d9ca0c78818c6f4c8ce: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/23476c4bcf6e4b4085d59d29739b6edb
I20240503 14:14:06.286799 30785 catalog_manager.cc:4830] TS e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297): tablet 23476c4bcf6e4b4085d59d29739b6edb (table rebalance_test_table_4 [id=f05910cae90c4382a503986593cff68f]) successfully deleted
I20240503 14:14:06.288766 30921 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:06.290036 31200 raft_consensus.cc:1261] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:06.290462 528 raft_consensus.cc:1261] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:06.290594 3098 consensus_queue.cc:1035] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:06.291114 3098 consensus_queue.cc:1035] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:06.291448 31180 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:06.299302 506 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:06.304596 792 raft_consensus.cc:1261] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5 [term 1 LEARNER]: Refusing update from remote peer 6dc4948ed7c740ab89b9e349d151b0e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:06.306227 3286 consensus_queue.cc:1035] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:06.329612 3098 raft_consensus.cc:2932] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 6dc4948ed7c740ab89b9e349d151b0e5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:06.338310 525 raft_consensus.cc:2932] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:06.340540 790 raft_consensus.cc:2932] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:06.350577 30796 catalog_manager.cc:5484] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91 reported cstate change: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:06.385972 31199 raft_consensus.cc:2932] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P e7badbe63a8a4d9ca0c78818c6f4c8ce [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f69eefe4294197923dff5efadcbbd5 (127.22.242.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } }
I20240503 14:14:06.404407 376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:06.414343 842 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 14:13:47.106794 845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:47.107348 845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:47.120013 845 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:50.078702 845 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 14:13:51.946327 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6d48b36155b41d3a75fe51d7d204ea7: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:51.946509 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:51.946722 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:51.946833 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:51.946941 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5b67f0a6a8d34286994597065f21a92b: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:51.947098 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:51.947197 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:51.947316 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:51.947410 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table acc4ce6b628f4f498f830c4572e30d65: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:51.947513 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:51.947661 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca6b9f7a0ded4c3aa1ea18aa2055c41b: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:51.947762 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5b67f0a6a8d34286994597065f21a92b: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:51.947845 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:51.947952 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:51.948072 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6d48b36155b41d3a75fe51d7d204ea7: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 4665a3377574417ba0d3165c236533c0
I20240503 14:13:52.133908 845 rebalancer_tool.cc:1074] tablet 63b2120788a54af78f02449f46bf23c7: '0dc902d91ccc435a82a5ede4a7729811' -> '85f69eefe4294197923dff5efadcbbd5' move scheduled
I20240503 14:13:52.253494 845 rebalancer_tool.cc:1084] tablet 1bb75d7d9499475ca60c633c0801d639: '6dc4948ed7c740ab89b9e349d151b0e5' -> '4665a3377574417ba0d3165c236533c0' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:52.253656 845 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:13:54.305814 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table acc4ce6b628f4f498f830c4572e30d65: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:54.305976 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:54.306080 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6d48b36155b41d3a75fe51d7d204ea7: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:54.306241 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:54.306344 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5b67f0a6a8d34286994597065f21a92b: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:54.364835 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:54.365038 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:54.365168 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:54.365265 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table acc4ce6b628f4f498f830c4572e30d65: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:54.365412 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:54.365506 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:54.365634 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca6b9f7a0ded4c3aa1ea18aa2055c41b: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:54.365790 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:54.365902 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:54.366050 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:54.366169 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca6b9f7a0ded4c3aa1ea18aa2055c41b: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:54.366259 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6d48b36155b41d3a75fe51d7d204ea7: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:54.366379 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:54.366462 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table acc4ce6b628f4f498f830c4572e30d65: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:54.366632 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server fa71cbdf0e544b81bd4e7e27196d2c91
I20240503 14:13:55.150801 845 rebalancer_tool.cc:1074] tablet ebd6dc710e144e61a5fae3911f25ee5c: 'e7badbe63a8a4d9ca0c78818c6f4c8ce' -> '4665a3377574417ba0d3165c236533c0' move scheduled
I20240503 14:13:55.476042 845 rebalancer_tool.cc:1084] tablet f6aeaa3fe1cd4127a6144d85b0434057: '82cfdc130877418dbe9d980d368973be' -> '6967bdda50404d68a5ac043b709440e1' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:55.476215 845 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:13:58.271435 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:58.271596 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:58.271729 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca6b9f7a0ded4c3aa1ea18aa2055c41b: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:58.271889 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5b67f0a6a8d34286994597065f21a92b: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:58.271977 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:58.272059 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:58.272197 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:58.272281 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:58.272365 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:58.272441 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca6b9f7a0ded4c3aa1ea18aa2055c41b: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:58.272549 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 6967bdda50404d68a5ac043b709440e1
I20240503 14:13:58.688087 845 rebalancer_tool.cc:1074] tablet c837def6d76242f08b20a448541e64d6: '6dc4948ed7c740ab89b9e349d151b0e5' -> '6967bdda50404d68a5ac043b709440e1' move scheduled
I20240503 14:13:58.881922 845 rebalancer_tool.cc:1074] tablet 2f82073122a74268a464387659dd90bf: '82cfdc130877418dbe9d980d368973be' -> 'fa71cbdf0e544b81bd4e7e27196d2c91' move scheduled
I20240503 14:13:59.305747 845 rebalancer_tool.cc:1074] tablet 55cf34de74694f47ad3f9f4d9bdffae4: '0dc902d91ccc435a82a5ede4a7729811' -> '85f69eefe4294197923dff5efadcbbd5' move scheduled
I20240503 14:13:59.420410 845 rebalancer_tool.cc:1074] tablet 626fcb185af0446b89ae82066e08981d: '82cfdc130877418dbe9d980d368973be' -> '4665a3377574417ba0d3165c236533c0' move scheduled
I20240503 14:13:59.515388 845 rebalancer_tool.cc:1084] tablet f3330e28c2d7434699b78070dd92ed73: 'e7badbe63a8a4d9ca0c78818c6f4c8ce' -> '6967bdda50404d68a5ac043b709440e1' move ignored: Illegal state: tablet f3330e28c2d7434699b78070dd92ed73: replica 6967bdda50404d68a5ac043b709440e1 is already present
I20240503 14:13:59.515563 845 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:14:01.884814 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:14:01.890048 845 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server fa71cbdf0e544b81bd4e7e27196d2c91
I20240503 14:14:02.191335 845 rebalancer_tool.cc:1074] tablet f3330e28c2d7434699b78070dd92ed73: '6dc4948ed7c740ab89b9e349d151b0e5' -> '' move scheduled
I20240503 14:14:02.471603 845 rebalancer_tool.cc:1074] tablet 1a9b84d307b8432bbdf6b70a11bf66b9: 'e7badbe63a8a4d9ca0c78818c6f4c8ce' -> '6967bdda50404d68a5ac043b709440e1' move scheduled
I20240503 14:14:02.653020 845 rebalancer_tool.cc:1084] tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87: '6dc4948ed7c740ab89b9e349d151b0e5' -> 'fa71cbdf0e544b81bd4e7e27196d2c91' move ignored: Illegal state: tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87: replica fa71cbdf0e544b81bd4e7e27196d2c91 is already present
I20240503 14:14:02.653251 845 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:14:04.535885 3378 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:14:04.005018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.242.71:46453 (local address 127.0.0.1:50288)
0503 14:14:04.029587 (+ 24569us) negotiation.cc:107] Waiting for socket to connect
0503 14:14:04.029608 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0503 14:14:04.029714 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:14:04.534462 (+504748us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":24455,"thread_start_us":26134,"threads_started":1}
Illegal state: tablet server 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): unacceptable health status UNAVAILABLE
I20240503 14:14:06.421849 3626 tablet_copy_client.cc:783] T da9dd5cd81114817b0f898c6b8fb3c07 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:06.422199 3626 tablet_copy_client.cc:647] T da9dd5cd81114817b0f898c6b8fb3c07 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:06.425815 3626 tablet_copy_client.cc:523] T da9dd5cd81114817b0f898c6b8fb3c07 P fa71cbdf0e544b81bd4e7e27196d2c91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:06.426026 31050 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:14:06.454214 3626 tablet_bootstrap.cc:492] T da9dd5cd81114817b0f898c6b8fb3c07 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap starting.
I20240503 14:14:06.465557 840 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 14:13:46.757319 849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:46.757906 849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:46.770243 849 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240503 14:13:49.469108 1208 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.22.242.126:34247, user_credentials={real_user=slave}} blocked reactor thread for 54118.6us
I20240503 14:13:50.278350 849 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 14:13:51.952311 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca6b9f7a0ded4c3aa1ea18aa2055c41b: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:51.952564 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6d48b36155b41d3a75fe51d7d204ea7: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:51.952805 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:51.952956 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:51.953183 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table acc4ce6b628f4f498f830c4572e30d65: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:51.953292 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:51.953382 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:51.953511 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:51.953620 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:51.953758 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5b67f0a6a8d34286994597065f21a92b: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:51.953845 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table acc4ce6b628f4f498f830c4572e30d65: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:51.953939 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:51.954022 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6d48b36155b41d3a75fe51d7d204ea7: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:51.954133 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 6967bdda50404d68a5ac043b709440e1
I20240503 14:13:52.067337 849 rebalancer_tool.cc:1074] tablet 2f82073122a74268a464387659dd90bf: '0dc902d91ccc435a82a5ede4a7729811' -> '85f69eefe4294197923dff5efadcbbd5' move scheduled
I20240503 14:13:52.285243 849 rebalancer_tool.cc:1074] tablet 1bb75d7d9499475ca60c633c0801d639: '6dc4948ed7c740ab89b9e349d151b0e5' -> '6967bdda50404d68a5ac043b709440e1' move scheduled
I20240503 14:13:52.495647 849 rebalancer_tool.cc:1074] tablet 43221a82a1124bf8840c6369f27f47f2: '82cfdc130877418dbe9d980d368973be' -> 'fa71cbdf0e544b81bd4e7e27196d2c91' move scheduled
I20240503 14:13:52.660387 849 rebalancer_tool.cc:1074] tablet 55cf34de74694f47ad3f9f4d9bdffae4: 'e7badbe63a8a4d9ca0c78818c6f4c8ce' -> '4665a3377574417ba0d3165c236533c0' move scheduled
I20240503 14:13:53.038161 849 rebalancer_tool.cc:1074] tablet 626fcb185af0446b89ae82066e08981d: '0dc902d91ccc435a82a5ede4a7729811' -> '85f69eefe4294197923dff5efadcbbd5' move scheduled
I20240503 14:13:53.586024 849 rebalancer_tool.cc:1074] tablet 1a9b84d307b8432bbdf6b70a11bf66b9: '6dc4948ed7c740ab89b9e349d151b0e5' -> 'fa71cbdf0e544b81bd4e7e27196d2c91' move scheduled
I20240503 14:13:53.948870 849 rebalancer_tool.cc:1084] tablet 63b2120788a54af78f02449f46bf23c7: '82cfdc130877418dbe9d980d368973be' -> '6967bdda50404d68a5ac043b709440e1' move ignored: Illegal state: tablet 63b2120788a54af78f02449f46bf23c7: replica 6967bdda50404d68a5ac043b709440e1 is already present
I20240503 14:13:53.949031 849 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:13:56.800976 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:56.801918 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:56.802163 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:56.805806 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:56.805984 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:56.806100 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca6b9f7a0ded4c3aa1ea18aa2055c41b: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:56.806205 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5b67f0a6a8d34286994597065f21a92b: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:56.806311 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:56.806458 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:56.806607 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:56.806723 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:56.806960 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:56.807077 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:56.807212 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:56.807307 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca6b9f7a0ded4c3aa1ea18aa2055c41b: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:56.807415 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5b67f0a6a8d34286994597065f21a92b: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:56.807516 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:56.807642 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table acc4ce6b628f4f498f830c4572e30d65: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:56.807725 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server fa71cbdf0e544b81bd4e7e27196d2c91
I20240503 14:13:57.334683 849 rebalancer_tool.cc:1074] tablet 5051c4e81ea14146a9f1d82b7f5b5f1d: '82cfdc130877418dbe9d980d368973be' -> '4665a3377574417ba0d3165c236533c0' move scheduled
I20240503 14:13:57.702772 849 rebalancer_tool.cc:1074] tablet f3330e28c2d7434699b78070dd92ed73: 'e7badbe63a8a4d9ca0c78818c6f4c8ce' -> '6967bdda50404d68a5ac043b709440e1' move scheduled
I20240503 14:13:57.832864 849 rebalancer_tool.cc:1084] tablet 589fce6d925b43d895588bac355135b0: '6dc4948ed7c740ab89b9e349d151b0e5' -> '85f69eefe4294197923dff5efadcbbd5' move ignored: Illegal state: tablet 589fce6d925b43d895588bac355135b0: replica 85f69eefe4294197923dff5efadcbbd5 is already present
I20240503 14:13:57.833039 849 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:13:59.780575 2552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.22.242.66:33221, user_credentials={real_user=slave}} blocked reactor thread for 65023us
W20240503 14:13:59.830410 2552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.22.242.69:42189, user_credentials={real_user=slave}} blocked reactor thread for 48807.1us
W20240503 14:14:00.392060 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:14:00.396304 849 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 6967bdda50404d68a5ac043b709440e1
I20240503 14:14:00.522465 849 rebalancer_tool.cc:1074] tablet d6b9a10c474f46bda0310ce661022b42: '0dc902d91ccc435a82a5ede4a7729811' -> '85f69eefe4294197923dff5efadcbbd5' move scheduled
I20240503 14:14:00.763034 849 rebalancer_tool.cc:1074] tablet f3e80559017d4a6ba16d34e0b7aab085: '6dc4948ed7c740ab89b9e349d151b0e5' -> '4665a3377574417ba0d3165c236533c0' move scheduled
I20240503 14:14:01.112671 849 rebalancer_tool.cc:1074] tablet f6aeaa3fe1cd4127a6144d85b0434057: '0dc902d91ccc435a82a5ede4a7729811' -> 'fa71cbdf0e544b81bd4e7e27196d2c91' move scheduled
I20240503 14:14:01.460711 849 rebalancer_tool.cc:1074] tablet 584769402a7e4e8a985a8f2363e2b8b9: '82cfdc130877418dbe9d980d368973be' -> '6967bdda50404d68a5ac043b709440e1' move scheduled
I20240503 14:14:01.903986 849 rebalancer_tool.cc:1074] tablet c86ee72320b647eb84e376f8995b55d2: 'e7badbe63a8a4d9ca0c78818c6f4c8ce' -> '85f69eefe4294197923dff5efadcbbd5' move scheduled
I20240503 14:14:02.087163 849 rebalancer_tool.cc:1074] tablet e7b5dd5eb46a45989bf9ffe3144bc189: '6dc4948ed7c740ab89b9e349d151b0e5' -> '4665a3377574417ba0d3165c236533c0' move scheduled
I20240503 14:14:02.419952 849 rebalancer_tool.cc:1074] tablet 3bd38c5cccf34dc3b8e6ec5425cb5e87: '6dc4948ed7c740ab89b9e349d151b0e5' -> 'fa71cbdf0e544b81bd4e7e27196d2c91' move scheduled
I20240503 14:14:02.607713 849 rebalancer_tool.cc:1074] tablet bc77a7da906b419085258f72667ff185: '0dc902d91ccc435a82a5ede4a7729811' -> '6967bdda50404d68a5ac043b709440e1' move scheduled
I20240503 14:14:02.921258 849 rebalancer_tool.cc:1074] tablet 23476c4bcf6e4b4085d59d29739b6edb: 'e7badbe63a8a4d9ca0c78818c6f4c8ce' -> '85f69eefe4294197923dff5efadcbbd5' move scheduled
I20240503 14:14:03.376569 849 rebalancer_tool.cc:1084] tablet 69b741aad35d4bfc8fea5b8e21d4a733: 'e7badbe63a8a4d9ca0c78818c6f4c8ce' -> '4665a3377574417ba0d3165c236533c0' move ignored: Illegal state: tablet 69b741aad35d4bfc8fea5b8e21d4a733: replica 4665a3377574417ba0d3165c236533c0 is already present
I20240503 14:14:03.376767 849 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): unacceptable health status UNAVAILABLE
I20240503 14:14:06.465979 844 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 14:13:47.176460 848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:47.177067 848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:47.209489 848 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:50.565729 848 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 14:13:52.256713 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6d48b36155b41d3a75fe51d7d204ea7: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:52.256906 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:52.257047 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:52.257174 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:52.257320 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5b67f0a6a8d34286994597065f21a92b: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:52.257447 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6d48b36155b41d3a75fe51d7d204ea7: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:52.257562 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table acc4ce6b628f4f498f830c4572e30d65: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:52.257716 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:52.257823 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:52.257958 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:52.258051 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:52.258157 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:52.258245 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca6b9f7a0ded4c3aa1ea18aa2055c41b: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:52.258419 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:52.258531 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table acc4ce6b628f4f498f830c4572e30d65: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:52.258643 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server fa71cbdf0e544b81bd4e7e27196d2c91
I20240503 14:13:52.398092 848 rebalancer_tool.cc:1074] tablet e7b5dd5eb46a45989bf9ffe3144bc189: 'e7badbe63a8a4d9ca0c78818c6f4c8ce' -> 'fa71cbdf0e544b81bd4e7e27196d2c91' move scheduled
I20240503 14:13:52.522728 848 rebalancer_tool.cc:1074] tablet 63b2120788a54af78f02449f46bf23c7: '0dc902d91ccc435a82a5ede4a7729811' -> '6967bdda50404d68a5ac043b709440e1' move scheduled
I20240503 14:13:52.725478 848 rebalancer_tool.cc:1084] tablet f17feb9b64214eaebcbe31dc45c75756: '82cfdc130877418dbe9d980d368973be' -> '85f69eefe4294197923dff5efadcbbd5' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: true } }
I20240503 14:13:52.725697 848 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:13:56.132006 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:56.132360 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:56.132484 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca6b9f7a0ded4c3aa1ea18aa2055c41b: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:56.132593 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5b67f0a6a8d34286994597065f21a92b: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:56.132683 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:56.132927 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5b67f0a6a8d34286994597065f21a92b: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:56.133044 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:56.133163 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6d48b36155b41d3a75fe51d7d204ea7: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:56.133337 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:56.133446 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:56.133544 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:56.133633 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca6b9f7a0ded4c3aa1ea18aa2055c41b: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:56.133733 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:56.133811 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5b67f0a6a8d34286994597065f21a92b: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 85f69eefe4294197923dff5efadcbbd5
I20240503 14:13:56.284646 848 rebalancer_tool.cc:1084] tablet ebd6dc710e144e61a5fae3911f25ee5c: '6dc4948ed7c740ab89b9e349d151b0e5' -> '4665a3377574417ba0d3165c236533c0' move ignored: Illegal state: tablet ebd6dc710e144e61a5fae3911f25ee5c: replica 4665a3377574417ba0d3165c236533c0 is already present
I20240503 14:13:56.284775 848 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:13:58.914698 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:58.916069 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server 4665a3377574417ba0d3165c236533c0 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:58.918263 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:58.918401 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca6b9f7a0ded4c3aa1ea18aa2055c41b: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:58.918514 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table acc4ce6b628f4f498f830c4572e30d65: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 6967bdda50404d68a5ac043b709440e1
I20240503 14:13:59.235423 848 rebalancer_tool.cc:1074] tablet ebd6dc710e144e61a5fae3911f25ee5c: '6dc4948ed7c740ab89b9e349d151b0e5' -> '85f69eefe4294197923dff5efadcbbd5' move scheduled
I20240503 14:13:59.383936 848 rebalancer_tool.cc:1084] tablet da9dd5cd81114817b0f898c6b8fb3c07: '82cfdc130877418dbe9d980d368973be' -> '4665a3377574417ba0d3165c236533c0' move ignored: Illegal state: tablet da9dd5cd81114817b0f898c6b8fb3c07: replica 4665a3377574417ba0d3165c236533c0 is already present
I20240503 14:13:59.384071 848 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:14:02.558565 848 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server fa71cbdf0e544b81bd4e7e27196d2c91
I20240503 14:14:03.006844 848 rebalancer_tool.cc:1074] tablet 69b741aad35d4bfc8fea5b8e21d4a733: '82cfdc130877418dbe9d980d368973be' -> '4665a3377574417ba0d3165c236533c0' move scheduled
I20240503 14:14:03.443259 848 rebalancer_tool.cc:1074] tablet f3330e28c2d7434699b78070dd92ed73: 'e7badbe63a8a4d9ca0c78818c6f4c8ce' -> '' move scheduled
I20240503 14:14:03.766860 848 rebalancer_tool.cc:1084] tablet bc77a7da906b419085258f72667ff185: '0dc902d91ccc435a82a5ede4a7729811' -> '6967bdda50404d68a5ac043b709440e1' move ignored: Illegal state: tablet bc77a7da906b419085258f72667ff185: replica 6967bdda50404d68a5ac043b709440e1 is already present
I20240503 14:14:03.766979 848 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): unacceptable health status UNAVAILABLE
I20240503 14:14:06.497606 3626 tablet_bootstrap.cc:492] T da9dd5cd81114817b0f898c6b8fb3c07 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:06.498034 3626 tablet_bootstrap.cc:492] T da9dd5cd81114817b0f898c6b8fb3c07 P fa71cbdf0e544b81bd4e7e27196d2c91: Bootstrap complete.
I20240503 14:14:06.498288 3626 ts_tablet_manager.cc:1367] T da9dd5cd81114817b0f898c6b8fb3c07 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent bootstrapping tablet: real 0.052s user 0.032s sys 0.004s
I20240503 14:14:06.498986 3626 raft_consensus.cc:348] T da9dd5cd81114817b0f898c6b8fb3c07 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:14:06.499222 3626 raft_consensus.cc:729] T da9dd5cd81114817b0f898c6b8fb3c07 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa71cbdf0e544b81bd4e7e27196d2c91, State: Initialized, Role: LEARNER
I20240503 14:14:06.499423 3626 consensus_queue.cc:260] T da9dd5cd81114817b0f898c6b8fb3c07 P fa71cbdf0e544b81bd4e7e27196d2c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: true } }
I20240503 14:14:06.499898 3626 ts_tablet_manager.cc:1398] T da9dd5cd81114817b0f898c6b8fb3c07 P fa71cbdf0e544b81bd4e7e27196d2c91: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 14:14:06.500700 31092 tablet_copy_service.cc:342] P 6dc4948ed7c740ab89b9e349d151b0e5: Request end of tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-da9dd5cd81114817b0f898c6b8fb3c07 received from {username='slave'} at 127.22.242.70:40181
I20240503 14:14:06.500831 31092 tablet_copy_service.cc:433] P 6dc4948ed7c740ab89b9e349d151b0e5: ending tablet copy session fa71cbdf0e544b81bd4e7e27196d2c91-da9dd5cd81114817b0f898c6b8fb3c07 on tablet da9dd5cd81114817b0f898c6b8fb3c07 with peer fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:14:06.580097 31138 consensus_peers.cc:485] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:14:06.647611 843 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 14:13:46.800521 846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:46.801069 846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:46.814100 846 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:50.423990 846 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 14:13:52.057991 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:52.058271 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table acc4ce6b628f4f498f830c4572e30d65: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:52.058387 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:52.058578 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:52.058670 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6d48b36155b41d3a75fe51d7d204ea7: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:52.058773 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:52.058867 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:52.059007 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:52.059130 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:52.059226 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca6b9f7a0ded4c3aa1ea18aa2055c41b: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:52.059316 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5b67f0a6a8d34286994597065f21a92b: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:52.059473 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table acc4ce6b628f4f498f830c4572e30d65: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:52.059576 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6d48b36155b41d3a75fe51d7d204ea7: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:52.059674 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 4665a3377574417ba0d3165c236533c0
I20240503 14:13:52.220057 846 rebalancer_tool.cc:1084] tablet 2f82073122a74268a464387659dd90bf: '0dc902d91ccc435a82a5ede4a7729811' -> '6967bdda50404d68a5ac043b709440e1' move ignored: Illegal state: tablet 2f82073122a74268a464387659dd90bf: replica 6967bdda50404d68a5ac043b709440e1 is already present
I20240503 14:13:52.220381 846 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:13:54.757756 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:54.757946 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6d48b36155b41d3a75fe51d7d204ea7: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:54.758095 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:54.758355 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:54.758478 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:54.758692 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:54.758814 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:54.758942 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca6b9f7a0ded4c3aa1ea18aa2055c41b: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:54.759091 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 026356e4d1494d1c8649c4b1d401565d: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:13:54.759254 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:54.759362 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca6b9f7a0ded4c3aa1ea18aa2055c41b: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:54.759485 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table acc4ce6b628f4f498f830c4572e30d65: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:54.759559 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:54.759722 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6d48b36155b41d3a75fe51d7d204ea7: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:54.759840 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server 4665a3377574417ba0d3165c236533c0
W20240503 14:13:54.759990 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:13:54.760116 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5b67f0a6a8d34286994597065f21a92b: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 6967bdda50404d68a5ac043b709440e1
W20240503 14:13:54.796862 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c243ced42cda4e3996901964941b50f4: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 85f69eefe4294197923dff5efadcbbd5
I20240503 14:13:55.488399 846 rebalancer_tool.cc:1074] tablet 5abcdc783c6144f28b9f8a9e1e162006: '0dc902d91ccc435a82a5ede4a7729811' -> '4665a3377574417ba0d3165c236533c0' move scheduled
I20240503 14:13:56.247082 846 rebalancer_tool.cc:1074] tablet 23476c4bcf6e4b4085d59d29739b6edb: '6dc4948ed7c740ab89b9e349d151b0e5' -> 'fa71cbdf0e544b81bd4e7e27196d2c91' move scheduled
I20240503 14:13:56.492406 846 rebalancer_tool.cc:1074] tablet 589fce6d925b43d895588bac355135b0: '82cfdc130877418dbe9d980d368973be' -> '85f69eefe4294197923dff5efadcbbd5' move scheduled
I20240503 14:13:56.657158 846 rebalancer_tool.cc:1074] tablet bc77a7da906b419085258f72667ff185: 'e7badbe63a8a4d9ca0c78818c6f4c8ce' -> '6967bdda50404d68a5ac043b709440e1' move scheduled
I20240503 14:13:56.914418 846 rebalancer_tool.cc:1074] tablet 4ff2fa69f79644c5b93beeba32fc9ca0: '0dc902d91ccc435a82a5ede4a7729811' -> '4665a3377574417ba0d3165c236533c0' move scheduled
I20240503 14:13:57.453130 846 rebalancer_tool.cc:1084] tablet ebd6dc710e144e61a5fae3911f25ee5c: '6dc4948ed7c740ab89b9e349d151b0e5' -> '4665a3377574417ba0d3165c236533c0' move ignored: Illegal state: tablet ebd6dc710e144e61a5fae3911f25ee5c: replica 4665a3377574417ba0d3165c236533c0 is already present
I20240503 14:13:57.453282 846 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:14:00.501168 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 85f69eefe4294197923dff5efadcbbd5
W20240503 14:14:00.501470 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04f8dfb749c04486895d6b0f7c206446: could not find any suitable replica to move from server 0dc902d91ccc435a82a5ede4a7729811 to server fa71cbdf0e544b81bd4e7e27196d2c91
W20240503 14:14:00.501638 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server 82cfdc130877418dbe9d980d368973be to server 0dc902d91ccc435a82a5ede4a7729811
W20240503 14:14:00.505640 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f05910cae90c4382a503986593cff68f: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 4665a3377574417ba0d3165c236533c0
I20240503 14:14:00.665422 846 rebalancer_tool.cc:1074] tablet 63b2120788a54af78f02449f46bf23c7: '6dc4948ed7c740ab89b9e349d151b0e5' -> '85f69eefe4294197923dff5efadcbbd5' move scheduled
I20240503 14:14:01.258674 846 rebalancer_tool.cc:1074] tablet 91a7b8a53140429b956178d43478a541: 'e7badbe63a8a4d9ca0c78818c6f4c8ce' -> '4665a3377574417ba0d3165c236533c0' move scheduled
I20240503 14:14:01.689222 846 rebalancer_tool.cc:1084] tablet f6aeaa3fe1cd4127a6144d85b0434057: '6dc4948ed7c740ab89b9e349d151b0e5' -> 'fa71cbdf0e544b81bd4e7e27196d2c91' move ignored: Illegal state: tablet f6aeaa3fe1cd4127a6144d85b0434057: replica fa71cbdf0e544b81bd4e7e27196d2c91 is already present
I20240503 14:14:01.689348 846 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:14:02.778858 3161 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.22.242.126:34247, user_credentials={real_user=slave}} blocked reactor thread for 58075.3us
W20240503 14:14:03.996397 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa8e781fa3e140b6865ce5f7b429e4c4: could not find any suitable replica to move from server e7badbe63a8a4d9ca0c78818c6f4c8ce to server 0dc902d91ccc435a82a5ede4a7729811
W20240503 14:14:04.004783 846 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08c2ccf24fcc40f9a0fd947d52b50f8e: could not find any suitable replica to move from server 6dc4948ed7c740ab89b9e349d151b0e5 to server fa71cbdf0e544b81bd4e7e27196d2c91
I20240503 14:14:05.175951 846 rebalancer_tool.cc:1074] tablet da9dd5cd81114817b0f898c6b8fb3c07: '0dc902d91ccc435a82a5ede4a7729811' -> '' move scheduled
I20240503 14:14:05.348659 846 rebalancer_tool.cc:1084] tablet 584769402a7e4e8a985a8f2363e2b8b9: '82cfdc130877418dbe9d980d368973be' -> '4665a3377574417ba0d3165c236533c0' move ignored: Not found: Peer with uuid 82cfdc130877418dbe9d980d368973be not found in consensus config
I20240503 14:14:05.348824 846 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): unacceptable health status UNAVAILABLE
E20240503 14:14:06.648257 23497 external_mini_cluster.cc:943] Process with UUID 4665a3377574417ba0d3165c236533c0 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 1
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240503 14:14:06.649699 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 30859
I20240503 14:14:06.658525 30981 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:14:06.661581 31268 consensus_peers.cc:462] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111)
W20240503 14:14:06.837127 31008 consensus_peers.cc:485] T e7b5dd5eb46a45989bf9ffe3144bc189 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:06.842240 31138 consensus_peers.cc:485] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:14:06.924751 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 30859
I20240503 14:14:06.964789 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 30990
W20240503 14:14:06.965090 31135 connection.cc:535] client connection to 127.22.242.65:46013 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:06.966008 31265 consensus_peers.cc:485] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be -> Peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): Couldn't send request to peer 0dc902d91ccc435a82a5ede4a7729811. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:46013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:06.966159 31265 consensus_peers.cc:485] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be -> Peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): Couldn't send request to peer 0dc902d91ccc435a82a5ede4a7729811. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:46013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:06.966245 31265 consensus_peers.cc:485] T bc77a7da906b419085258f72667ff185 P 82cfdc130877418dbe9d980d368973be -> Peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): Couldn't send request to peer 0dc902d91ccc435a82a5ede4a7729811. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:46013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:06.966285 31135 consensus_peers.cc:485] T dd5869fd8f8d41d99cf531edb7df3750 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): Couldn't send request to peer 0dc902d91ccc435a82a5ede4a7729811. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:46013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:06.966462 31135 consensus_peers.cc:485] T f3330e28c2d7434699b78070dd92ed73 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): Couldn't send request to peer 0dc902d91ccc435a82a5ede4a7729811. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:46013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:06.966557 31135 consensus_peers.cc:485] T 43221a82a1124bf8840c6369f27f47f2 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): Couldn't send request to peer 0dc902d91ccc435a82a5ede4a7729811. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:46013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:06.966703 31005 consensus_peers.cc:485] T da9dd5cd81114817b0f898c6b8fb3c07 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): Couldn't send request to peer 0dc902d91ccc435a82a5ede4a7729811. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:46013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:06.966846 31005 consensus_peers.cc:485] T 589fce6d925b43d895588bac355135b0 P 6dc4948ed7c740ab89b9e349d151b0e5 -> Peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): Couldn't send request to peer 0dc902d91ccc435a82a5ede4a7729811. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:46013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:06.970935 31112 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:14:07.122234 31268 consensus_peers.cc:485] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:14:07.240778 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 30990
W20240503 14:14:07.249101 31268 consensus_peers.cc:462] T 69b741aad35d4bfc8fea5b8e21d4a733 P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111)
W20240503 14:14:07.287727 31135 consensus_peers.cc:485] T 91a7b8a53140429b956178d43478a541 P e7badbe63a8a4d9ca0c78818c6f4c8ce -> Peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): Couldn't send request to peer 0dc902d91ccc435a82a5ede4a7729811. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:46013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:07.291543 31265 connection.cc:535] server connection from 127.22.242.66:48159 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:07.291962 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 31120
I20240503 14:14:07.305264 31239 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:14:07.330152 31268 consensus_peers.cc:485] T 55cf34de74694f47ad3f9f4d9bdffae4 P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:07.500104 31268 consensus_peers.cc:485] T 6234489254fc4882a39b7df28ddf673f P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:07.523833 31268 consensus_peers.cc:485] T 626fcb185af0446b89ae82066e08981d P 82cfdc130877418dbe9d980d368973be -> Peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221): Couldn't send request to peer 6dc4948ed7c740ab89b9e349d151b0e5. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.66:33221: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:07.556236 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 31120
W20240503 14:14:07.559527 31268 consensus_peers.cc:485] T 0f45824b1fff4302abd03c1fb48c6c7b P 82cfdc130877418dbe9d980d368973be -> Peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Couldn't send request to peer 4665a3377574417ba0d3165c236533c0. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:07.597223 30784 connection.cc:535] server connection from 127.22.242.67:59969 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:07.597246 331 connection.cc:535] server connection from 127.22.242.67:32969 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:07.597602 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 31250
I20240503 14:14:07.603416 31371 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:07.621438 3698 raft_consensus.cc:482] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0dc902d91ccc435a82a5ede4a7729811)
I20240503 14:14:07.621711 3698 raft_consensus.cc:504] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:14:07.623625 3698 leader_election.cc:288] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221), fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829)
W20240503 14:14:07.624795 332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.242.66:33221: connect: Connection refused (error 111)
W20240503 14:14:07.628849 332 leader_election.cc:334] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221): Network error: Client connection negotiation failed: client connection to 127.22.242.66:33221: connect: Connection refused (error 111)
W20240503 14:14:07.629503 329 leader_election.cc:334] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): Network error: Client connection negotiation failed: client connection to 127.22.242.65:46013: connect: Connection refused (error 111)
I20240503 14:14:07.629621 329 leader_election.cc:302] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6967bdda50404d68a5ac043b709440e1; no voters: 0dc902d91ccc435a82a5ede4a7729811, 6dc4948ed7c740ab89b9e349d151b0e5
I20240503 14:14:07.629985 3698 raft_consensus.cc:2726] T f6aeaa3fe1cd4127a6144d85b0434057 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:14:07.633785 525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6aeaa3fe1cd4127a6144d85b0434057" candidate_uuid: "6967bdda50404d68a5ac043b709440e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" is_pre_election: true
I20240503 14:14:07.696136 3704 raft_consensus.cc:482] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0dc902d91ccc435a82a5ede4a7729811)
I20240503 14:14:07.696425 3704 raft_consensus.cc:504] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:14:07.698052 3704 leader_election.cc:288] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189)
W20240503 14:14:07.711308 461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.242.65:46013: connect: Connection refused (error 111)
W20240503 14:14:07.715653 461 leader_election.cc:334] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): Network error: Client connection negotiation failed: client connection to 127.22.242.65:46013: connect: Connection refused (error 111)
I20240503 14:14:07.721452 393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a9b84d307b8432bbdf6b70a11bf66b9" candidate_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "6967bdda50404d68a5ac043b709440e1" is_pre_election: true
I20240503 14:14:07.722290 464 leader_election.cc:302] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa71cbdf0e544b81bd4e7e27196d2c91; no voters: 0dc902d91ccc435a82a5ede4a7729811, 6967bdda50404d68a5ac043b709440e1
I20240503 14:14:07.722726 3704 raft_consensus.cc:2726] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:14:07.853878 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 31250
I20240503 14:14:07.893384 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 314
W20240503 14:14:07.893437 725 connection.cc:535] server connection from 127.22.242.68:48217 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:07.899051 432 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:07.986217 3704 raft_consensus.cc:482] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dc4948ed7c740ab89b9e349d151b0e5)
I20240503 14:14:07.986457 3704 raft_consensus.cc:504] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:14:07.987368 3704 leader_election.cc:288] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221), 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189)
W20240503 14:14:07.989251 464 leader_election.cc:334] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221): Network error: Client connection negotiation failed: client connection to 127.22.242.66:33221: connect: Connection refused (error 111)
I20240503 14:14:08.084298 3698 raft_consensus.cc:482] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dc4948ed7c740ab89b9e349d151b0e5)
I20240503 14:14:08.084527 3698 raft_consensus.cc:504] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:14:08.084697 393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584769402a7e4e8a985a8f2363e2b8b9" candidate_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "6967bdda50404d68a5ac043b709440e1" is_pre_election: true
I20240503 14:14:08.085425 3698 leader_election.cc:288] T c837def6d76242f08b20a448541e64d6 P 6967bdda50404d68a5ac043b709440e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221), e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829)
I20240503 14:14:08.085518 464 leader_election.cc:302] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa71cbdf0e544b81bd4e7e27196d2c91; no voters: 6967bdda50404d68a5ac043b709440e1, 6dc4948ed7c740ab89b9e349d151b0e5
I20240503 14:14:08.085599 3698 raft_consensus.cc:482] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0dc902d91ccc435a82a5ede4a7729811)
I20240503 14:14:08.085747 3698 raft_consensus.cc:504] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } }
I20240503 14:14:08.085988 3704 raft_consensus.cc:2726] T 584769402a7e4e8a985a8f2363e2b8b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:14:08.086325 3698 leader_election.cc:288] T 1a9b84d307b8432bbdf6b70a11bf66b9 P 6967bdda50404d68a5ac043b709440e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829)
I20240503 14:14:08.086567 3704 raft_consensus.cc:482] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0dc902d91ccc435a82a5ede4a7729811)
I20240503 14:14:08.086740 3704 raft_consensus.cc:504] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } attrs { replace: true } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } }
I20240503 14:14:08.087663 3704 leader_election.cc:288] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221), 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189)
I20240503 14:14:08.087692 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 314
I20240503 14:14:08.086843 525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c837def6d76242f08b20a448541e64d6" candidate_uuid: "6967bdda50404d68a5ac043b709440e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" is_pre_election: true
I20240503 14:14:08.087610 528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a9b84d307b8432bbdf6b70a11bf66b9" candidate_uuid: "6967bdda50404d68a5ac043b709440e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" is_pre_election: true
I20240503 14:14:08.089751 528 raft_consensus.cc:2454] T 1a9b84d307b8432bbdf6b70a11bf66b9 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6967bdda50404d68a5ac043b709440e1 in term 1.
W20240503 14:14:08.090283 461 leader_election.cc:334] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): Network error: Client connection negotiation failed: client connection to 127.22.242.65:46013: connect: Connection refused (error 111)
W20240503 14:14:08.091202 464 leader_election.cc:334] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221): Network error: Client connection negotiation failed: client connection to 127.22.242.66:33221: connect: Connection refused (error 111)
I20240503 14:14:08.091329 464 leader_election.cc:302] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fa71cbdf0e544b81bd4e7e27196d2c91; no voters: 0dc902d91ccc435a82a5ede4a7729811, 6dc4948ed7c740ab89b9e349d151b0e5
I20240503 14:14:08.091565 3704 raft_consensus.cc:2726] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:14:08.120609 464 connection.cc:535] client connection to 127.22.242.69:42189 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:08.120783 462 connection.cc:535] server connection from 127.22.242.69:35863 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:08.120945 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 445
W20240503 14:14:08.121992 464 leader_election.cc:334] T f6aeaa3fe1cd4127a6144d85b0434057 P fa71cbdf0e544b81bd4e7e27196d2c91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189): Network error: Client connection negotiation failed: client connection to 127.22.242.69:42189: connect: Connection refused (error 111)
I20240503 14:14:08.126688 568 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:08.169507 3714 raft_consensus.cc:482] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dc4948ed7c740ab89b9e349d151b0e5)
I20240503 14:14:08.169788 3714 raft_consensus.cc:504] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:08.171437 3714 leader_election.cc:288] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221), 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453)
W20240503 14:14:08.173298 725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.242.65:46013: connect: Connection refused (error 111)
W20240503 14:14:08.174840 725 leader_election.cc:334] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): Network error: Client connection negotiation failed: client connection to 127.22.242.65:46013: connect: Connection refused (error 111)
W20240503 14:14:08.175134 728 leader_election.cc:334] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221): Network error: Client connection negotiation failed: client connection to 127.22.242.66:33221: connect: Connection refused (error 111)
I20240503 14:14:08.175268 728 leader_election.cc:302] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85f69eefe4294197923dff5efadcbbd5; no voters: 0dc902d91ccc435a82a5ede4a7729811, 6dc4948ed7c740ab89b9e349d151b0e5
I20240503 14:14:08.175611 3714 raft_consensus.cc:2726] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:14:08.175637 728 leader_election.cc:334] T ebd6dc710e144e61a5fae3911f25ee5c P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111)
I20240503 14:14:08.204475 3714 raft_consensus.cc:482] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dc4948ed7c740ab89b9e349d151b0e5)
I20240503 14:14:08.204700 3714 raft_consensus.cc:504] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:08.205452 3714 leader_election.cc:288] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221), e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297)
W20240503 14:14:08.206928 728 leader_election.cc:334] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221): Network error: Client connection negotiation failed: client connection to 127.22.242.66:33221: connect: Connection refused (error 111)
W20240503 14:14:08.207126 726 leader_election.cc:334] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297): Network error: Client connection negotiation failed: client connection to 127.22.242.67:44297: connect: Connection refused (error 111)
I20240503 14:14:08.207250 726 leader_election.cc:302] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f69eefe4294197923dff5efadcbbd5; no voters: 6dc4948ed7c740ab89b9e349d151b0e5, e7badbe63a8a4d9ca0c78818c6f4c8ce
I20240503 14:14:08.207535 3714 raft_consensus.cc:2726] T d6b9a10c474f46bda0310ce661022b42 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:14:08.239243 3714 raft_consensus.cc:482] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dc4948ed7c740ab89b9e349d151b0e5)
I20240503 14:14:08.239441 3714 raft_consensus.cc:504] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dc902d91ccc435a82a5ede4a7729811" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 46013 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:08.240327 3714 leader_election.cc:288] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221)
W20240503 14:14:08.241868 728 leader_election.cc:334] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221): Network error: Client connection negotiation failed: client connection to 127.22.242.66:33221: connect: Connection refused (error 111)
W20240503 14:14:08.242086 725 leader_election.cc:334] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0dc902d91ccc435a82a5ede4a7729811 (127.22.242.65:46013): Network error: Client connection negotiation failed: client connection to 127.22.242.65:46013: connect: Connection refused (error 111)
I20240503 14:14:08.242228 725 leader_election.cc:302] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f69eefe4294197923dff5efadcbbd5; no voters: 0dc902d91ccc435a82a5ede4a7729811, 6dc4948ed7c740ab89b9e349d151b0e5
I20240503 14:14:08.242456 3714 raft_consensus.cc:2726] T 589fce6d925b43d895588bac355135b0 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:14:08.254035 3714 raft_consensus.cc:482] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dc4948ed7c740ab89b9e349d151b0e5)
I20240503 14:14:08.254230 3714 raft_consensus.cc:504] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:08.255332 3714 leader_election.cc:288] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221), fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829)
W20240503 14:14:08.256503 726 leader_election.cc:334] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e7badbe63a8a4d9ca0c78818c6f4c8ce (127.22.242.67:44297): Network error: Client connection negotiation failed: client connection to 127.22.242.67:44297: connect: Connection refused (error 111)
W20240503 14:14:08.256904 728 leader_election.cc:334] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221): Network error: Client connection negotiation failed: client connection to 127.22.242.66:33221: connect: Connection refused (error 111)
I20240503 14:14:08.257030 728 leader_election.cc:302] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85f69eefe4294197923dff5efadcbbd5; no voters: 6dc4948ed7c740ab89b9e349d151b0e5, e7badbe63a8a4d9ca0c78818c6f4c8ce
I20240503 14:14:08.257295 3714 raft_consensus.cc:2726] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:14:08.283713 3714 raft_consensus.cc:482] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dc4948ed7c740ab89b9e349d151b0e5)
I20240503 14:14:08.283918 3714 raft_consensus.cc:504] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } } peers { permanent_uuid: "4665a3377574417ba0d3165c236533c0" member_type: VOTER last_known_addr { host: "127.22.242.71" port: 46453 } attrs { promote: false } }
I20240503 14:14:08.284889 3714 leader_election.cc:288] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221), 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453)
W20240503 14:14:08.286582 727 leader_election.cc:334] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063): Network error: Client connection negotiation failed: client connection to 127.22.242.68:37063: connect: Connection refused (error 111)
W20240503 14:14:08.286748 728 leader_election.cc:334] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221): Network error: Client connection negotiation failed: client connection to 127.22.242.66:33221: connect: Connection refused (error 111)
I20240503 14:14:08.286890 728 leader_election.cc:302] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85f69eefe4294197923dff5efadcbbd5; no voters: 6dc4948ed7c740ab89b9e349d151b0e5, 82cfdc130877418dbe9d980d368973be
I20240503 14:14:08.287127 3714 raft_consensus.cc:2726] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:14:08.287163 728 leader_election.cc:334] T f3e80559017d4a6ba16d34e0b7aab085 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4665a3377574417ba0d3165c236533c0 (127.22.242.71:46453): Network error: Client connection negotiation failed: client connection to 127.22.242.71:46453: connect: Connection refused (error 111)
I20240503 14:14:08.298053 3714 raft_consensus.cc:482] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dc4948ed7c740ab89b9e349d151b0e5)
I20240503 14:14:08.298240 3714 raft_consensus.cc:504] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "82cfdc130877418dbe9d980d368973be" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 37063 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:08.299134 3714 leader_election.cc:288] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221), 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063), 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189)
W20240503 14:14:08.300531 727 leader_election.cc:334] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82cfdc130877418dbe9d980d368973be (127.22.242.68:37063): Network error: Client connection negotiation failed: client connection to 127.22.242.68:37063: connect: Connection refused (error 111)
W20240503 14:14:08.300724 728 leader_election.cc:334] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6dc4948ed7c740ab89b9e349d151b0e5 (127.22.242.66:33221): Network error: Client connection negotiation failed: client connection to 127.22.242.66:33221: connect: Connection refused (error 111)
I20240503 14:14:08.300848 728 leader_election.cc:302] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 85f69eefe4294197923dff5efadcbbd5; no voters: 6dc4948ed7c740ab89b9e349d151b0e5, 82cfdc130877418dbe9d980d368973be
I20240503 14:14:08.301085 3714 raft_consensus.cc:2726] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:14:08.301142 728 leader_election.cc:334] T 63b2120788a54af78f02449f46bf23c7 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6967bdda50404d68a5ac043b709440e1 (127.22.242.69:42189): Network error: Client connection negotiation failed: client connection to 127.22.242.69:42189: connect: Connection refused (error 111)
I20240503 14:14:08.320371 3704 raft_consensus.cc:482] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dc4948ed7c740ab89b9e349d151b0e5)
I20240503 14:14:08.320617 3704 raft_consensus.cc:504] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P fa71cbdf0e544b81bd4e7e27196d2c91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7badbe63a8a4d9ca0c78818c6f4c8ce" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44297 } } peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } attrs { replace: true } } peers { permanent_uuid: "fa71cbdf0e544b81bd4e7e27196d2c91" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 46829 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:08.321380 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 445
I20240503 14:14:08.352757 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 708
W20240503 14:14:08.352901 3717 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:14:08.255240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.242.70:46829 (local address 127.22.242.72:53551)
0503 14:14:08.255758 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0503 14:14:08.255773 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0503 14:14:08.255899 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:14:08.352525 (+ 96626us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.242.70:46829: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":39}
W20240503 14:14:08.354168 725 leader_election.cc:334] T 3bd38c5cccf34dc3b8e6ec5425cb5e87 P 85f69eefe4294197923dff5efadcbbd5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fa71cbdf0e544b81bd4e7e27196d2c91 (127.22.242.70:46829): Network error: Client connection negotiation failed: client connection to 127.22.242.70:46829: connect: Connection refused (error 111)
I20240503 14:14:08.359162 832 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:08.546159 3714 raft_consensus.cc:482] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dc4948ed7c740ab89b9e349d151b0e5)
I20240503 14:14:08.546375 3714 raft_consensus.cc:504] T c86ee72320b647eb84e376f8995b55d2 P 85f69eefe4294197923dff5efadcbbd5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6dc4948ed7c740ab89b9e349d151b0e5" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 33221 } } peers { permanent_uuid: "6967bdda50404d68a5ac043b709440e1" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42189 } attrs { promote: false } } peers { permanent_uuid: "85f69eefe4294197923dff5efadcbbd5" member_type: VOTER last_known_addr { host: "127.22.242.72" port: 36545 } attrs { promote: false } }
I20240503 14:14:08.546666 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 708
I20240503 14:14:08.578475 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 30769
I20240503 14:14:08.579553 30829 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:08.818804 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 30769
2024-05-03T14:14:08Z chronyd exiting
I20240503 14:14:08.878353 23497 test_util.cc:175] -----------------------------------------------
I20240503 14:14:08.878438 23497 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714745515241910-23497-0
[ FAILED ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1, where GetParam() = 4-byte object <01-00 00-00> (47641 ms)
[----------] 1 test from ConcurrentRebalancersTest (47641 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1
I20240503 14:14:08.880705 23497 test_util.cc:268] Using random seed: 1430789160
I20240503 14:14:08.882473 23497 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:14:08.882539 23497 ts_itest-base.cc:113] --------------
I20240503 14:14:08.882575 23497 ts_itest-base.cc:114] 4 tablet servers
I20240503 14:14:08.882611 23497 ts_itest-base.cc:115] 1 replicas per TS
I20240503 14:14:08.882649 23497 ts_itest-base.cc:116] --------------
2024-05-03T14:14:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:14:08Z Disabled control of system clock
I20240503 14:14:08.928503 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.242.126:40059
--webserver_interface=127.22.242.126
--webserver_port=0
--builtin_ntp_servers=127.22.242.84:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.242.126:40059
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240503 14:14:09.261579 3729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:09.261874 3729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:09.261989 3729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:09.269238 3729 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:09.269354 3729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:14:09.269434 3729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:09.269515 3729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:14:09.269593 3729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:14:09.278420 3729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.242.126:40059
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.242.126:40059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.242.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:09.279706 3729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:09.281471 3729 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:09.292026 3734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:09.292037 3735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:09.292698 3737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:09.292876 3729 server_base.cc:1034] running on GCE node
I20240503 14:14:09.293710 3729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:09.295136 3729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:09.296331 3729 hybrid_clock.cc:648] HybridClock initialized: now 1714745649296285 us; error 52 us; skew 500 ppm
I20240503 14:14:09.296659 3729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:09.298947 3729 webserver.cc:460] Webserver started at http://127.22.242.126:39359/ using document root <none> and password file <none>
I20240503 14:14:09.299676 3729 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:09.299798 3729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:09.300107 3729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:09.302677 3729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "606ba1dfda514492a9c4398a3acebca4"
format_stamp: "Formatted at 2024-05-03 14:14:09 on dist-test-slave-r3pd"
I20240503 14:14:09.303468 3729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "606ba1dfda514492a9c4398a3acebca4"
format_stamp: "Formatted at 2024-05-03 14:14:09 on dist-test-slave-r3pd"
I20240503 14:14:09.308818 3729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 14:14:09.312192 3743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:09.313503 3729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:14:09.313666 3729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
uuid: "606ba1dfda514492a9c4398a3acebca4"
format_stamp: "Formatted at 2024-05-03 14:14:09 on dist-test-slave-r3pd"
I20240503 14:14:09.313860 3729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:09.329044 3729 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:09.329814 3729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:09.330081 3729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:09.347152 3729 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.126:40059
I20240503 14:14:09.347172 3794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.126:40059 every 8 connection(s)
I20240503 14:14:09.348507 3729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:14:09.351943 3795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:09.354933 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 3729
I20240503 14:14:09.355134 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:14:09.363232 3795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4: Bootstrap starting.
I20240503 14:14:09.366676 3795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:09.368054 3795 log.cc:824] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:09.370661 3795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4: No bootstrap required, opened a new log
I20240503 14:14:09.376487 3795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606ba1dfda514492a9c4398a3acebca4" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 40059 } }
I20240503 14:14:09.376813 3795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:09.376916 3795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606ba1dfda514492a9c4398a3acebca4, State: Initialized, Role: FOLLOWER
I20240503 14:14:09.377513 3795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606ba1dfda514492a9c4398a3acebca4" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 40059 } }
I20240503 14:14:09.377717 3795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:09.377835 3795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:09.377982 3795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:09.380018 3795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606ba1dfda514492a9c4398a3acebca4" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 40059 } }
I20240503 14:14:09.380518 3795 leader_election.cc:302] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 606ba1dfda514492a9c4398a3acebca4; no voters:
I20240503 14:14:09.381157 3795 leader_election.cc:288] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:14:09.381290 3800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:09.381927 3800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4 [term 1 LEADER]: Becoming Leader. State: Replica: 606ba1dfda514492a9c4398a3acebca4, State: Running, Role: LEADER
I20240503 14:14:09.382514 3800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606ba1dfda514492a9c4398a3acebca4" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 40059 } }
I20240503 14:14:09.383127 3795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:14:09.385696 3801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "606ba1dfda514492a9c4398a3acebca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606ba1dfda514492a9c4398a3acebca4" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 40059 } } }
I20240503 14:14:09.385636 3802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 606ba1dfda514492a9c4398a3acebca4. Latest consensus state: current_term: 1 leader_uuid: "606ba1dfda514492a9c4398a3acebca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606ba1dfda514492a9c4398a3acebca4" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 40059 } } }
I20240503 14:14:09.386157 3801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4 [sys.catalog]: This master's current role is: LEADER
I20240503 14:14:09.386181 3802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4 [sys.catalog]: This master's current role is: LEADER
I20240503 14:14:09.387064 3809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:14:09.392488 3809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:14:09.400099 3809 catalog_manager.cc:1347] Generated new cluster ID: 9dc35a4e944c460780644d101229f9b3
I20240503 14:14:09.400242 3809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:14:09.423990 3809 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:14:09.425271 3809 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:14:09.434648 3809 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 606ba1dfda514492a9c4398a3acebca4: Generated new TSK 0
I20240503 14:14:09.435642 3809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:14:09.451893 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.65:0
--local_ip_for_outbound_sockets=127.22.242.65
--webserver_interface=127.22.242.65
--webserver_port=0
--tserver_master_addrs=127.22.242.126:40059
--builtin_ntp_servers=127.22.242.84:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:14:09.813751 3819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:09.814052 3819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:09.814182 3819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:09.821930 3819 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:09.822060 3819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:09.822232 3819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.65
I20240503 14:14:09.831472 3819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.242.65
--webserver_port=0
--tserver_master_addrs=127.22.242.126:40059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.65
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:09.832813 3819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:09.834650 3819 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:09.845919 3825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:09.845947 3827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:09.845996 3824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:09.846956 3819 server_base.cc:1034] running on GCE node
I20240503 14:14:09.847673 3819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:09.848632 3819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:09.849821 3819 hybrid_clock.cc:648] HybridClock initialized: now 1714745649849776 us; error 36 us; skew 500 ppm
I20240503 14:14:09.850138 3819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:09.852255 3819 webserver.cc:460] Webserver started at http://127.22.242.65:41617/ using document root <none> and password file <none>
I20240503 14:14:09.852986 3819 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:09.853106 3819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:09.853413 3819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:09.855952 3819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "fb22456acd9240fcbdbd570531f9a10b"
format_stamp: "Formatted at 2024-05-03 14:14:09 on dist-test-slave-r3pd"
I20240503 14:14:09.856716 3819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "fb22456acd9240fcbdbd570531f9a10b"
format_stamp: "Formatted at 2024-05-03 14:14:09 on dist-test-slave-r3pd"
I20240503 14:14:09.861500 3819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20240503 14:14:09.864619 3833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:09.865841 3819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240503 14:14:09.865983 3819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fb22456acd9240fcbdbd570531f9a10b"
format_stamp: "Formatted at 2024-05-03 14:14:09 on dist-test-slave-r3pd"
I20240503 14:14:09.866166 3819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:09.879874 3819 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:09.880746 3819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:09.880999 3819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:09.881939 3819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:09.883381 3819 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:09.883479 3819 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:09.883589 3819 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:09.883685 3819 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:09.915546 3819 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.65:36431
I20240503 14:14:09.915616 3945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.65:36431 every 8 connection(s)
I20240503 14:14:09.916848 3819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:14:09.926872 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 3819
I20240503 14:14:09.927083 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:14:09.929915 3946 heartbeater.cc:347] Connected to a master server at 127.22.242.126:40059
I20240503 14:14:09.930310 3946 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:09.930467 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.66:0
--local_ip_for_outbound_sockets=127.22.242.66
--webserver_interface=127.22.242.66
--webserver_port=0
--tserver_master_addrs=127.22.242.126:40059
--builtin_ntp_servers=127.22.242.84:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:14:09.931347 3946 heartbeater.cc:511] Master 127.22.242.126:40059 requested a full tablet report, sending...
I20240503 14:14:09.933524 3760 ts_manager.cc:194] Registered new tserver with Master: fb22456acd9240fcbdbd570531f9a10b (127.22.242.65:36431)
I20240503 14:14:09.935051 3760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.65:36387
W20240503 14:14:10.269747 3950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:10.270049 3950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:10.270185 3950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:10.277580 3950 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:10.277709 3950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:10.277825 3950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.66
I20240503 14:14:10.286906 3950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.242.66
--webserver_port=0
--tserver_master_addrs=127.22.242.126:40059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.66
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:10.288287 3950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:10.290022 3950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:10.301924 3956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:10.301883 3955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:10.302498 3958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:10.302642 3950 server_base.cc:1034] running on GCE node
I20240503 14:14:10.303280 3950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:10.304260 3950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:10.305466 3950 hybrid_clock.cc:648] HybridClock initialized: now 1714745650305405 us; error 42 us; skew 500 ppm
I20240503 14:14:10.305789 3950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:10.308281 3950 webserver.cc:460] Webserver started at http://127.22.242.66:32915/ using document root <none> and password file <none>
I20240503 14:14:10.309192 3950 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:10.309302 3950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:10.309581 3950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:10.312120 3950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8f38339f4d434e39ba4cbc9cc0d236f0"
format_stamp: "Formatted at 2024-05-03 14:14:10 on dist-test-slave-r3pd"
I20240503 14:14:10.312907 3950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8f38339f4d434e39ba4cbc9cc0d236f0"
format_stamp: "Formatted at 2024-05-03 14:14:10 on dist-test-slave-r3pd"
I20240503 14:14:10.318070 3950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:14:10.321446 3964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:10.322840 3950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:14:10.323012 3950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8f38339f4d434e39ba4cbc9cc0d236f0"
format_stamp: "Formatted at 2024-05-03 14:14:10 on dist-test-slave-r3pd"
I20240503 14:14:10.323211 3950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:10.336593 3950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:10.337404 3950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:10.337669 3950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:10.338594 3950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:10.340092 3950 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:10.340193 3950 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:10.340294 3950 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:10.340360 3950 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:10.374095 3950 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.66:46329
I20240503 14:14:10.374145 4076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.66:46329 every 8 connection(s)
I20240503 14:14:10.375347 3950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:14:10.380548 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 3950
I20240503 14:14:10.380705 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:14:10.384275 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.67:0
--local_ip_for_outbound_sockets=127.22.242.67
--webserver_interface=127.22.242.67
--webserver_port=0
--tserver_master_addrs=127.22.242.126:40059
--builtin_ntp_servers=127.22.242.84:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:14:10.388749 4077 heartbeater.cc:347] Connected to a master server at 127.22.242.126:40059
I20240503 14:14:10.389156 4077 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:10.390087 4077 heartbeater.cc:511] Master 127.22.242.126:40059 requested a full tablet report, sending...
I20240503 14:14:10.391498 3760 ts_manager.cc:194] Registered new tserver with Master: 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66:46329)
I20240503 14:14:10.392313 3760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.66:56017
W20240503 14:14:10.721807 4080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:10.722118 4080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:10.722256 4080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:10.729817 4080 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:10.729948 4080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:10.730141 4080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.67
I20240503 14:14:10.739114 4080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.242.67
--webserver_port=0
--tserver_master_addrs=127.22.242.126:40059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.67
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:10.740505 4080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:10.742266 4080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:10.754029 4086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:10.754061 4088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:10.754128 4085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:10.755939 4080 server_base.cc:1034] running on GCE node
I20240503 14:14:10.756543 4080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:10.757522 4080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:10.758756 4080 hybrid_clock.cc:648] HybridClock initialized: now 1714745650758706 us; error 44 us; skew 500 ppm
I20240503 14:14:10.759116 4080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:10.761328 4080 webserver.cc:460] Webserver started at http://127.22.242.67:40551/ using document root <none> and password file <none>
I20240503 14:14:10.762040 4080 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:10.762163 4080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:10.762462 4080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:10.765164 4080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f105cb873c7b4f7a90ae1ee6088947c0"
format_stamp: "Formatted at 2024-05-03 14:14:10 on dist-test-slave-r3pd"
I20240503 14:14:10.765997 4080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f105cb873c7b4f7a90ae1ee6088947c0"
format_stamp: "Formatted at 2024-05-03 14:14:10 on dist-test-slave-r3pd"
I20240503 14:14:10.771200 4080 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240503 14:14:10.774621 4094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:10.775941 4080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:10.776084 4080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f105cb873c7b4f7a90ae1ee6088947c0"
format_stamp: "Formatted at 2024-05-03 14:14:10 on dist-test-slave-r3pd"
I20240503 14:14:10.776312 4080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:10.792271 4080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:10.793094 4080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:10.793375 4080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:10.794338 4080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:10.795809 4080 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:10.795895 4080 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:10.796108 4080 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:10.796223 4080 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240503 14:14:10.828621 4080 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.67:33985
I20240503 14:14:10.828666 4206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.67:33985 every 8 connection(s)
I20240503 14:14:10.829959 4080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:14:10.834105 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 4080
I20240503 14:14:10.834286 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:14:10.837810 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.68:0
--local_ip_for_outbound_sockets=127.22.242.68
--webserver_interface=127.22.242.68
--webserver_port=0
--tserver_master_addrs=127.22.242.126:40059
--builtin_ntp_servers=127.22.242.84:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:14:10.843964 4207 heartbeater.cc:347] Connected to a master server at 127.22.242.126:40059
I20240503 14:14:10.844328 4207 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:10.845340 4207 heartbeater.cc:511] Master 127.22.242.126:40059 requested a full tablet report, sending...
I20240503 14:14:10.846891 3760 ts_manager.cc:194] Registered new tserver with Master: f105cb873c7b4f7a90ae1ee6088947c0 (127.22.242.67:33985)
I20240503 14:14:10.847682 3760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.67:36711
I20240503 14:14:10.938395 3946 heartbeater.cc:503] Master 127.22.242.126:40059 was elected leader, sending a full tablet report...
W20240503 14:14:11.175065 4210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:11.175380 4210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:11.175529 4210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:11.183102 4210 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:11.183252 4210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:11.183442 4210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.68
I20240503 14:14:11.192464 4210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.242.68
--webserver_port=0
--tserver_master_addrs=127.22.242.126:40059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.68
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:11.193783 4210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:11.195503 4210 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:11.207139 4216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:11.207104 4218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:11.207191 4215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:11.207993 4210 server_base.cc:1034] running on GCE node
I20240503 14:14:11.208622 4210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:11.209574 4210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:11.210796 4210 hybrid_clock.cc:648] HybridClock initialized: now 1714745651210734 us; error 58 us; skew 500 ppm
I20240503 14:14:11.211212 4210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:11.213646 4210 webserver.cc:460] Webserver started at http://127.22.242.68:43177/ using document root <none> and password file <none>
I20240503 14:14:11.214375 4210 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:11.214502 4210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:11.214807 4210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:11.217453 4210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3734b96f895b4d45b2ce758377826d43"
format_stamp: "Formatted at 2024-05-03 14:14:11 on dist-test-slave-r3pd"
I20240503 14:14:11.218274 4210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3734b96f895b4d45b2ce758377826d43"
format_stamp: "Formatted at 2024-05-03 14:14:11 on dist-test-slave-r3pd"
I20240503 14:14:11.224399 4210 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240503 14:14:11.227705 4224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:11.229139 4210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:11.229295 4210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3734b96f895b4d45b2ce758377826d43"
format_stamp: "Formatted at 2024-05-03 14:14:11 on dist-test-slave-r3pd"
I20240503 14:14:11.229485 4210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:11.247169 4210 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:11.248016 4210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:11.248324 4210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:11.249244 4210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:11.250674 4210 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:11.250772 4210 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:11.250886 4210 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:11.250958 4210 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:11.281958 4210 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.68:38509
I20240503 14:14:11.282060 4336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.68:38509 every 8 connection(s)
I20240503 14:14:11.283267 4210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:14:11.287590 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 4210
I20240503 14:14:11.287782 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:14:11.296056 4337 heartbeater.cc:347] Connected to a master server at 127.22.242.126:40059
I20240503 14:14:11.296367 4337 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:11.297129 4337 heartbeater.cc:511] Master 127.22.242.126:40059 requested a full tablet report, sending...
I20240503 14:14:11.298381 3760 ts_manager.cc:194] Registered new tserver with Master: 3734b96f895b4d45b2ce758377826d43 (127.22.242.68:38509)
I20240503 14:14:11.299147 3760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.68:33515
I20240503 14:14:11.303542 23497 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 14:14:11.318945 3760 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60324:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 14:14:11.345239 4142 tablet_service.cc:1467] Processing CreateTablet for tablet 80912b19b76b43e4b6a6fe7d2294ae06 (DEFAULT_TABLE table=TestTable [id=69b111c8d57547a58fe5802acf2f97b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:11.346398 4142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80912b19b76b43e4b6a6fe7d2294ae06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:11.359473 4351 tablet_bootstrap.cc:492] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0: Bootstrap starting.
I20240503 14:14:11.362390 4351 tablet_bootstrap.cc:654] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:11.363394 4351 log.cc:824] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:11.365801 4351 tablet_bootstrap.cc:492] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0: No bootstrap required, opened a new log
I20240503 14:14:11.366063 4351 ts_tablet_manager.cc:1367] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:14:11.371780 4351 raft_consensus.cc:348] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } }
I20240503 14:14:11.372092 4351 raft_consensus.cc:374] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:11.372223 4351 raft_consensus.cc:729] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f105cb873c7b4f7a90ae1ee6088947c0, State: Initialized, Role: FOLLOWER
I20240503 14:14:11.372790 4351 consensus_queue.cc:260] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } }
I20240503 14:14:11.372979 4351 raft_consensus.cc:388] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:11.373133 4351 raft_consensus.cc:482] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:11.373289 4351 raft_consensus.cc:3037] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:11.376003 4351 raft_consensus.cc:504] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } }
I20240503 14:14:11.376574 4351 leader_election.cc:302] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f105cb873c7b4f7a90ae1ee6088947c0; no voters:
I20240503 14:14:11.377244 4351 leader_election.cc:288] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:14:11.377419 4353 raft_consensus.cc:2781] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:11.378468 4207 heartbeater.cc:503] Master 127.22.242.126:40059 was elected leader, sending a full tablet report...
I20240503 14:14:11.378791 4351 ts_tablet_manager.cc:1398] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20240503 14:14:11.379314 4353 raft_consensus.cc:686] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 LEADER]: Becoming Leader. State: Replica: f105cb873c7b4f7a90ae1ee6088947c0, State: Running, Role: LEADER
I20240503 14:14:11.379711 4353 consensus_queue.cc:237] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } }
I20240503 14:14:11.385370 3760 catalog_manager.cc:5484] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f105cb873c7b4f7a90ae1ee6088947c0 (127.22.242.67). New cstate: current_term: 1 leader_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:11.394749 4077 heartbeater.cc:503] Master 127.22.242.126:40059 was elected leader, sending a full tablet report...
I20240503 14:14:11.401952 23497 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 14:14:11.403993 23497 ts_itest-base.cc:233] Waiting for 0 tablets on tserver fb22456acd9240fcbdbd570531f9a10b to finish bootstrapping
I20240503 14:14:11.413427 23497 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8f38339f4d434e39ba4cbc9cc0d236f0 to finish bootstrapping
I20240503 14:14:11.422753 23497 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f105cb873c7b4f7a90ae1ee6088947c0 to finish bootstrapping
I20240503 14:14:11.430387 23497 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3734b96f895b4d45b2ce758377826d43 to finish bootstrapping
I20240503 14:14:11.439287 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 4080
I20240503 14:14:11.440270 4202 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:11.524260 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 4080
I20240503 14:14:11.541254 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 4210
I20240503 14:14:11.542558 4332 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:11.623353 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 4210
I20240503 14:14:15.391680 3760 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60324:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:14:15.411698 4012 tablet_service.cc:1467] Processing CreateTablet for tablet ec9f52d1b5744db68351657a42a78a5b (DEFAULT_TABLE table=rebalance_test_table_0 [id=42b5ab15f2f048179c8e0c1a6c99d22f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:15.413024 4012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec9f52d1b5744db68351657a42a78a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:15.412511 3880 tablet_service.cc:1467] Processing CreateTablet for tablet e5fc12cdadca4372b83ea547ca6daf4d (DEFAULT_TABLE table=rebalance_test_table_0 [id=42b5ab15f2f048179c8e0c1a6c99d22f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:15.412549 3881 tablet_service.cc:1467] Processing CreateTablet for tablet 3b1f3f7a061b40918d9666f19e3ccfdb (DEFAULT_TABLE table=rebalance_test_table_0 [id=42b5ab15f2f048179c8e0c1a6c99d22f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:15.414040 3880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5fc12cdadca4372b83ea547ca6daf4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:15.414510 3881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b1f3f7a061b40918d9666f19e3ccfdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:15.425601 4373 tablet_bootstrap.cc:492] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0: Bootstrap starting.
I20240503 14:14:15.425730 4372 tablet_bootstrap.cc:492] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b: Bootstrap starting.
I20240503 14:14:15.429251 4372 tablet_bootstrap.cc:654] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:15.429251 4373 tablet_bootstrap.cc:654] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:15.430596 4372 log.cc:824] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:15.430608 4373 log.cc:824] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:15.433537 4373 tablet_bootstrap.cc:492] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0: No bootstrap required, opened a new log
I20240503 14:14:15.433766 4372 tablet_bootstrap.cc:492] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b: No bootstrap required, opened a new log
I20240503 14:14:15.433817 4373 ts_tablet_manager.cc:1367] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 14:14:15.434024 4372 ts_tablet_manager.cc:1367] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20240503 14:14:15.443365 4372 raft_consensus.cc:348] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:15.443365 4373 raft_consensus.cc:348] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:15.443806 4372 raft_consensus.cc:374] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:15.443806 4373 raft_consensus.cc:374] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:15.443928 4372 raft_consensus.cc:729] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb22456acd9240fcbdbd570531f9a10b, State: Initialized, Role: FOLLOWER
I20240503 14:14:15.443923 4373 raft_consensus.cc:729] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f38339f4d434e39ba4cbc9cc0d236f0, State: Initialized, Role: FOLLOWER
I20240503 14:14:15.444708 4373 consensus_queue.cc:260] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:15.444715 4372 consensus_queue.cc:260] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:15.444978 4373 raft_consensus.cc:388] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:15.444978 4372 raft_consensus.cc:388] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:15.445127 4372 raft_consensus.cc:482] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:15.445127 4373 raft_consensus.cc:482] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:15.445338 4373 raft_consensus.cc:3037] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:15.445336 4372 raft_consensus.cc:3037] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:15.448805 4373 raft_consensus.cc:504] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:15.448805 4372 raft_consensus.cc:504] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:15.449400 4372 leader_election.cc:302] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb22456acd9240fcbdbd570531f9a10b; no voters:
I20240503 14:14:15.449400 4373 leader_election.cc:302] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f38339f4d434e39ba4cbc9cc0d236f0; no voters:
I20240503 14:14:15.450157 4373 leader_election.cc:288] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:14:15.450628 4372 leader_election.cc:288] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:14:15.450681 4377 raft_consensus.cc:2781] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:15.451076 4376 raft_consensus.cc:2781] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:15.456209 4377 raft_consensus.cc:686] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Becoming Leader. State: Replica: 8f38339f4d434e39ba4cbc9cc0d236f0, State: Running, Role: LEADER
I20240503 14:14:15.456372 4372 ts_tablet_manager.cc:1398] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b: Time spent starting tablet: real 0.022s user 0.019s sys 0.005s
I20240503 14:14:15.456957 4372 tablet_bootstrap.cc:492] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b: Bootstrap starting.
I20240503 14:14:15.456826 4377 consensus_queue.cc:237] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:15.459308 4372 tablet_bootstrap.cc:654] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:15.459998 4376 raft_consensus.cc:686] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Becoming Leader. State: Replica: fb22456acd9240fcbdbd570531f9a10b, State: Running, Role: LEADER
I20240503 14:14:15.460608 4376 consensus_queue.cc:237] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:15.461804 4373 ts_tablet_manager.cc:1398] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0: Time spent starting tablet: real 0.028s user 0.017s sys 0.008s
I20240503 14:14:15.464378 4372 tablet_bootstrap.cc:492] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b: No bootstrap required, opened a new log
I20240503 14:14:15.464602 4372 ts_tablet_manager.cc:1367] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240503 14:14:15.465652 4372 raft_consensus.cc:348] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:15.465891 4372 raft_consensus.cc:374] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:15.466050 4372 raft_consensus.cc:729] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb22456acd9240fcbdbd570531f9a10b, State: Initialized, Role: FOLLOWER
I20240503 14:14:15.466393 4372 consensus_queue.cc:260] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:15.466660 4372 raft_consensus.cc:388] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:15.466841 4372 raft_consensus.cc:482] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:15.466665 3760 catalog_manager.cc:5484] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b reported cstate change: term changed from 0 to 1, leader changed from <none> to fb22456acd9240fcbdbd570531f9a10b (127.22.242.65). New cstate: current_term: 1 leader_uuid: "fb22456acd9240fcbdbd570531f9a10b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:15.467128 4372 raft_consensus.cc:3037] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:15.468636 3759 catalog_manager.cc:5484] T ec9f52d1b5744db68351657a42a78a5b P 8f38339f4d434e39ba4cbc9cc0d236f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:15.471351 4372 raft_consensus.cc:504] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:15.471745 4372 leader_election.cc:302] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb22456acd9240fcbdbd570531f9a10b; no voters:
I20240503 14:14:15.472045 4372 leader_election.cc:288] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:14:15.472393 4379 raft_consensus.cc:2781] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:15.478495 4372 ts_tablet_manager.cc:1398] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b: Time spent starting tablet: real 0.014s user 0.003s sys 0.003s
I20240503 14:14:15.479275 4379 raft_consensus.cc:686] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Becoming Leader. State: Replica: fb22456acd9240fcbdbd570531f9a10b, State: Running, Role: LEADER
I20240503 14:14:15.479610 4379 consensus_queue.cc:237] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:15.483456 3760 catalog_manager.cc:5484] T 3b1f3f7a061b40918d9666f19e3ccfdb P fb22456acd9240fcbdbd570531f9a10b reported cstate change: term changed from 0 to 1, leader changed from <none> to fb22456acd9240fcbdbd570531f9a10b (127.22.242.65). New cstate: current_term: 1 leader_uuid: "fb22456acd9240fcbdbd570531f9a10b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:16.001456 3760 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60324:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:14:16.009406 4011 tablet_service.cc:1467] Processing CreateTablet for tablet 7885ec95faa045eeb65e4de68819b840 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0615440c9e1b4b49a0cfbc6d828ef5f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:16.009455 4012 tablet_service.cc:1467] Processing CreateTablet for tablet 1798538a1d0e46b7b8f95d275afe4662 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0615440c9e1b4b49a0cfbc6d828ef5f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:16.009765 3881 tablet_service.cc:1467] Processing CreateTablet for tablet 30216842696e4463a801892b43f5f77c (DEFAULT_TABLE table=rebalance_test_table_1 [id=0615440c9e1b4b49a0cfbc6d828ef5f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:16.009987 4011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7885ec95faa045eeb65e4de68819b840. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:16.010182 3881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30216842696e4463a801892b43f5f77c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:16.010218 4012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1798538a1d0e46b7b8f95d275afe4662. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:16.015779 4412 tablet_bootstrap.cc:492] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0: Bootstrap starting.
I20240503 14:14:16.017457 4412 tablet_bootstrap.cc:654] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:16.017547 4413 tablet_bootstrap.cc:492] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b: Bootstrap starting.
I20240503 14:14:16.019173 4413 tablet_bootstrap.cc:654] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:16.019455 4412 tablet_bootstrap.cc:492] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0: No bootstrap required, opened a new log
I20240503 14:14:16.019573 4412 ts_tablet_manager.cc:1367] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:14:16.020491 4412 raft_consensus.cc:348] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:16.020637 4412 raft_consensus.cc:374] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:16.020705 4412 raft_consensus.cc:729] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f38339f4d434e39ba4cbc9cc0d236f0, State: Initialized, Role: FOLLOWER
I20240503 14:14:16.020874 4413 tablet_bootstrap.cc:492] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b: No bootstrap required, opened a new log
I20240503 14:14:16.020927 4412 consensus_queue.cc:260] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:16.021018 4413 ts_tablet_manager.cc:1367] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:14:16.021099 4412 raft_consensus.cc:388] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:16.021178 4412 raft_consensus.cc:482] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:16.021291 4412 raft_consensus.cc:3037] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:16.021701 4413 raft_consensus.cc:348] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:16.021904 4413 raft_consensus.cc:374] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:16.022012 4413 raft_consensus.cc:729] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb22456acd9240fcbdbd570531f9a10b, State: Initialized, Role: FOLLOWER
I20240503 14:14:16.022243 4413 consensus_queue.cc:260] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:16.022444 4413 raft_consensus.cc:388] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:16.022547 4413 raft_consensus.cc:482] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:16.022661 4413 raft_consensus.cc:3037] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:16.023387 4412 raft_consensus.cc:504] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:16.023610 4412 leader_election.cc:302] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f38339f4d434e39ba4cbc9cc0d236f0; no voters:
I20240503 14:14:16.023779 4412 leader_election.cc:288] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:14:16.023866 4377 raft_consensus.cc:2781] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:16.024585 4412 ts_tablet_manager.cc:1398] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 14:14:16.024852 4412 tablet_bootstrap.cc:492] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0: Bootstrap starting.
I20240503 14:14:16.025133 4413 raft_consensus.cc:504] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:16.025470 4413 leader_election.cc:302] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb22456acd9240fcbdbd570531f9a10b; no voters:
I20240503 14:14:16.025993 4413 leader_election.cc:288] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:14:16.026221 4418 raft_consensus.cc:2781] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:16.026294 4412 tablet_bootstrap.cc:654] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:16.030370 4377 raft_consensus.cc:686] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Becoming Leader. State: Replica: 8f38339f4d434e39ba4cbc9cc0d236f0, State: Running, Role: LEADER
I20240503 14:14:16.030761 4377 consensus_queue.cc:237] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:16.031884 4412 tablet_bootstrap.cc:492] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0: No bootstrap required, opened a new log
I20240503 14:14:16.032080 4412 ts_tablet_manager.cc:1367] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 14:14:16.033103 4412 raft_consensus.cc:348] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:16.033331 4412 raft_consensus.cc:374] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:16.033429 4412 raft_consensus.cc:729] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f38339f4d434e39ba4cbc9cc0d236f0, State: Initialized, Role: FOLLOWER
I20240503 14:14:16.033818 4412 consensus_queue.cc:260] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:16.034040 4412 raft_consensus.cc:388] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:16.034170 4412 raft_consensus.cc:482] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:16.034292 4412 raft_consensus.cc:3037] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:16.038123 4412 raft_consensus.cc:504] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:16.038442 4412 leader_election.cc:302] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f38339f4d434e39ba4cbc9cc0d236f0; no voters:
I20240503 14:14:16.038715 4412 leader_election.cc:288] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:14:16.038789 4417 raft_consensus.cc:2781] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:16.039266 4412 ts_tablet_manager.cc:1398] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20240503 14:14:16.039681 4418 raft_consensus.cc:686] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Becoming Leader. State: Replica: fb22456acd9240fcbdbd570531f9a10b, State: Running, Role: LEADER
I20240503 14:14:16.040130 4418 consensus_queue.cc:237] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:16.042130 4413 ts_tablet_manager.cc:1398] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b: Time spent starting tablet: real 0.021s user 0.006s sys 0.014s
I20240503 14:14:16.042507 4417 raft_consensus.cc:686] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Becoming Leader. State: Replica: 8f38339f4d434e39ba4cbc9cc0d236f0, State: Running, Role: LEADER
I20240503 14:14:16.042806 4417 consensus_queue.cc:237] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:16.043967 3760 catalog_manager.cc:5484] T 1798538a1d0e46b7b8f95d275afe4662 P 8f38339f4d434e39ba4cbc9cc0d236f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:16.050078 3760 catalog_manager.cc:5484] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b reported cstate change: term changed from 0 to 1, leader changed from <none> to fb22456acd9240fcbdbd570531f9a10b (127.22.242.65). New cstate: current_term: 1 leader_uuid: "fb22456acd9240fcbdbd570531f9a10b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:16.051496 3759 catalog_manager.cc:5484] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:17.512065 3760 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60324:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:14:17.520804 3881 tablet_service.cc:1467] Processing CreateTablet for tablet 0825740e9f794e25ab75b2b14e90ea96 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4cecb121529244ab8fc498a140f28970]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:17.520974 4011 tablet_service.cc:1467] Processing CreateTablet for tablet baad63a6a5b64c3c952900c382c9f809 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4cecb121529244ab8fc498a140f28970]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:17.521296 3881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0825740e9f794e25ab75b2b14e90ea96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:17.521404 4011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baad63a6a5b64c3c952900c382c9f809. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:17.521667 3880 tablet_service.cc:1467] Processing CreateTablet for tablet 816ac90d991a4a7b9f1af35b95848900 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4cecb121529244ab8fc498a140f28970]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:17.522020 3880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816ac90d991a4a7b9f1af35b95848900. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:17.526398 4453 tablet_bootstrap.cc:492] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0: Bootstrap starting.
I20240503 14:14:17.528290 4453 tablet_bootstrap.cc:654] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:17.528627 4454 tablet_bootstrap.cc:492] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b: Bootstrap starting.
I20240503 14:14:17.529805 4454 tablet_bootstrap.cc:654] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:17.529958 4453 tablet_bootstrap.cc:492] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0: No bootstrap required, opened a new log
I20240503 14:14:17.530110 4453 ts_tablet_manager.cc:1367] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:14:17.531216 4453 raft_consensus.cc:348] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:17.531361 4453 raft_consensus.cc:374] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:17.531440 4454 tablet_bootstrap.cc:492] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b: No bootstrap required, opened a new log
I20240503 14:14:17.531455 4453 raft_consensus.cc:729] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f38339f4d434e39ba4cbc9cc0d236f0, State: Initialized, Role: FOLLOWER
I20240503 14:14:17.531567 4454 ts_tablet_manager.cc:1367] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:17.531682 4453 consensus_queue.cc:260] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:17.531854 4453 raft_consensus.cc:388] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:17.531936 4453 raft_consensus.cc:482] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:17.532042 4453 raft_consensus.cc:3037] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:17.532131 4454 raft_consensus.cc:348] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:17.532317 4454 raft_consensus.cc:374] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:17.532395 4454 raft_consensus.cc:729] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb22456acd9240fcbdbd570531f9a10b, State: Initialized, Role: FOLLOWER
I20240503 14:14:17.532645 4454 consensus_queue.cc:260] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:17.532795 4454 raft_consensus.cc:388] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:17.532879 4454 raft_consensus.cc:482] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:17.532982 4454 raft_consensus.cc:3037] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:17.533991 4453 raft_consensus.cc:504] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:17.534230 4453 leader_election.cc:302] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f38339f4d434e39ba4cbc9cc0d236f0; no voters:
I20240503 14:14:17.534683 4453 leader_election.cc:288] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:14:17.534881 4454 raft_consensus.cc:504] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:17.534842 4457 raft_consensus.cc:2781] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:17.535099 4454 leader_election.cc:302] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb22456acd9240fcbdbd570531f9a10b; no voters:
I20240503 14:14:17.535660 4457 raft_consensus.cc:686] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Becoming Leader. State: Replica: 8f38339f4d434e39ba4cbc9cc0d236f0, State: Running, Role: LEADER
I20240503 14:14:17.536082 4457 consensus_queue.cc:237] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:17.536500 4454 leader_election.cc:288] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:14:17.536644 4459 raft_consensus.cc:2781] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:17.542704 4453 ts_tablet_manager.cc:1398] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20240503 14:14:17.543609 4454 ts_tablet_manager.cc:1398] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20240503 14:14:17.543881 4454 tablet_bootstrap.cc:492] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b: Bootstrap starting.
I20240503 14:14:17.544991 4459 raft_consensus.cc:686] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Becoming Leader. State: Replica: fb22456acd9240fcbdbd570531f9a10b, State: Running, Role: LEADER
I20240503 14:14:17.545291 4459 consensus_queue.cc:237] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:17.545518 4454 tablet_bootstrap.cc:654] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:17.545388 3760 catalog_manager.cc:5484] T baad63a6a5b64c3c952900c382c9f809 P 8f38339f4d434e39ba4cbc9cc0d236f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:17.548795 3759 catalog_manager.cc:5484] T 816ac90d991a4a7b9f1af35b95848900 P fb22456acd9240fcbdbd570531f9a10b reported cstate change: term changed from 0 to 1, leader changed from <none> to fb22456acd9240fcbdbd570531f9a10b (127.22.242.65). New cstate: current_term: 1 leader_uuid: "fb22456acd9240fcbdbd570531f9a10b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:17.549510 4454 tablet_bootstrap.cc:492] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b: No bootstrap required, opened a new log
I20240503 14:14:17.549705 4454 ts_tablet_manager.cc:1367] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 14:14:17.551088 4454 raft_consensus.cc:348] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:17.551292 4454 raft_consensus.cc:374] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:17.551359 4454 raft_consensus.cc:729] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb22456acd9240fcbdbd570531f9a10b, State: Initialized, Role: FOLLOWER
I20240503 14:14:17.551599 4454 consensus_queue.cc:260] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:17.551777 4454 raft_consensus.cc:388] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:17.551875 4454 raft_consensus.cc:482] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:17.551977 4454 raft_consensus.cc:3037] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:17.554558 4454 raft_consensus.cc:504] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:17.554778 4454 leader_election.cc:302] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb22456acd9240fcbdbd570531f9a10b; no voters:
I20240503 14:14:17.555013 4454 leader_election.cc:288] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:14:17.555049 4459 raft_consensus.cc:2781] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:17.555230 4459 raft_consensus.cc:686] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Becoming Leader. State: Replica: fb22456acd9240fcbdbd570531f9a10b, State: Running, Role: LEADER
I20240503 14:14:17.555555 4454 ts_tablet_manager.cc:1398] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 14:14:17.555518 4459 consensus_queue.cc:237] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:17.557668 3760 catalog_manager.cc:5484] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b reported cstate change: term changed from 0 to 1, leader changed from <none> to fb22456acd9240fcbdbd570531f9a10b (127.22.242.65). New cstate: current_term: 1 leader_uuid: "fb22456acd9240fcbdbd570531f9a10b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:18.057889 3760 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60324:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:14:18.066473 4011 tablet_service.cc:1467] Processing CreateTablet for tablet a0e57448a9404822aae4a81fa9cefba7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d8bc56b41046428cbe11ffc734ef8e75]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:18.066573 4012 tablet_service.cc:1467] Processing CreateTablet for tablet b9d129cb330a4c1e9341798b08b55094 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d8bc56b41046428cbe11ffc734ef8e75]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:18.066767 3880 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3ea6cfa9714f89ba973d355e74ffab (DEFAULT_TABLE table=rebalance_test_table_3 [id=d8bc56b41046428cbe11ffc734ef8e75]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:18.066994 4011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e57448a9404822aae4a81fa9cefba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:18.067175 3880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3ea6cfa9714f89ba973d355e74ffab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:18.067198 4012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d129cb330a4c1e9341798b08b55094. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:18.072999 4494 tablet_bootstrap.cc:492] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0: Bootstrap starting.
I20240503 14:14:18.074591 4493 tablet_bootstrap.cc:492] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b: Bootstrap starting.
I20240503 14:14:18.074741 4494 tablet_bootstrap.cc:654] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:18.076009 4493 tablet_bootstrap.cc:654] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:18.076437 4494 tablet_bootstrap.cc:492] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0: No bootstrap required, opened a new log
I20240503 14:14:18.076592 4494 ts_tablet_manager.cc:1367] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:14:18.077193 4494 raft_consensus.cc:348] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:18.077337 4494 raft_consensus.cc:374] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:18.077431 4494 raft_consensus.cc:729] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f38339f4d434e39ba4cbc9cc0d236f0, State: Initialized, Role: FOLLOWER
I20240503 14:14:18.077642 4494 consensus_queue.cc:260] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:18.077785 4494 raft_consensus.cc:388] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:18.077838 4493 tablet_bootstrap.cc:492] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b: No bootstrap required, opened a new log
I20240503 14:14:18.077867 4494 raft_consensus.cc:482] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:18.077980 4494 raft_consensus.cc:3037] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:18.077972 4493 ts_tablet_manager.cc:1367] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:18.078532 4493 raft_consensus.cc:348] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:18.078670 4493 raft_consensus.cc:374] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:18.078765 4493 raft_consensus.cc:729] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb22456acd9240fcbdbd570531f9a10b, State: Initialized, Role: FOLLOWER
I20240503 14:14:18.078989 4493 consensus_queue.cc:260] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:18.079149 4493 raft_consensus.cc:388] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:18.079227 4493 raft_consensus.cc:482] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:18.079336 4493 raft_consensus.cc:3037] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:18.080008 4494 raft_consensus.cc:504] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:18.080255 4494 leader_election.cc:302] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f38339f4d434e39ba4cbc9cc0d236f0; no voters:
I20240503 14:14:18.080785 4494 leader_election.cc:288] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:14:18.080936 4497 raft_consensus.cc:2781] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:18.081985 4494 ts_tablet_manager.cc:1398] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 14:14:18.082163 4493 raft_consensus.cc:504] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:18.082378 4494 tablet_bootstrap.cc:492] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0: Bootstrap starting.
I20240503 14:14:18.082396 4493 leader_election.cc:302] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb22456acd9240fcbdbd570531f9a10b; no voters:
I20240503 14:14:18.082580 4493 leader_election.cc:288] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:14:18.082665 4459 raft_consensus.cc:2781] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:18.084702 4494 tablet_bootstrap.cc:654] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:18.086081 4459 raft_consensus.cc:686] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Becoming Leader. State: Replica: fb22456acd9240fcbdbd570531f9a10b, State: Running, Role: LEADER
I20240503 14:14:18.086402 4459 consensus_queue.cc:237] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } }
I20240503 14:14:18.088066 4497 raft_consensus.cc:686] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Becoming Leader. State: Replica: 8f38339f4d434e39ba4cbc9cc0d236f0, State: Running, Role: LEADER
I20240503 14:14:18.088542 4497 consensus_queue.cc:237] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:18.089891 4493 ts_tablet_manager.cc:1398] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b: Time spent starting tablet: real 0.012s user 0.005s sys 0.006s
I20240503 14:14:18.090692 4494 tablet_bootstrap.cc:492] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0: No bootstrap required, opened a new log
I20240503 14:14:18.090869 4494 ts_tablet_manager.cc:1367] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240503 14:14:18.091601 4494 raft_consensus.cc:348] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:18.092465 3760 catalog_manager.cc:5484] T b9d129cb330a4c1e9341798b08b55094 P 8f38339f4d434e39ba4cbc9cc0d236f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:18.092115 4494 raft_consensus.cc:374] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:18.093053 4494 raft_consensus.cc:729] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f38339f4d434e39ba4cbc9cc0d236f0, State: Initialized, Role: FOLLOWER
I20240503 14:14:18.094295 3758 catalog_manager.cc:5484] T 2a3ea6cfa9714f89ba973d355e74ffab P fb22456acd9240fcbdbd570531f9a10b reported cstate change: term changed from 0 to 1, leader changed from <none> to fb22456acd9240fcbdbd570531f9a10b (127.22.242.65). New cstate: current_term: 1 leader_uuid: "fb22456acd9240fcbdbd570531f9a10b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:18.096722 4494 consensus_queue.cc:260] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:18.097579 4494 raft_consensus.cc:388] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:18.097741 4494 raft_consensus.cc:482] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:18.097873 4494 raft_consensus.cc:3037] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:18.100428 4494 raft_consensus.cc:504] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:18.100661 4494 leader_election.cc:302] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f38339f4d434e39ba4cbc9cc0d236f0; no voters:
I20240503 14:14:18.100867 4494 leader_election.cc:288] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:14:18.100917 4497 raft_consensus.cc:2781] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:18.101944 4497 raft_consensus.cc:686] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Becoming Leader. State: Replica: 8f38339f4d434e39ba4cbc9cc0d236f0, State: Running, Role: LEADER
I20240503 14:14:18.102214 4494 ts_tablet_manager.cc:1398] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0: Time spent starting tablet: real 0.011s user 0.001s sys 0.004s
I20240503 14:14:18.102260 4497 consensus_queue.cc:237] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } }
I20240503 14:14:18.104416 3760 catalog_manager.cc:5484] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66). New cstate: current_term: 1 leader_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:18.601325 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.67:33985
--local_ip_for_outbound_sockets=127.22.242.67
--tserver_master_addrs=127.22.242.126:40059
--webserver_port=40551
--webserver_interface=127.22.242.67
--builtin_ntp_servers=127.22.242.84:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:14:18.940606 4528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:18.940887 4528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:18.940974 4528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:18.948284 4528 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:18.948395 4528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:18.948509 4528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.67
I20240503 14:14:18.957183 4528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.67:33985
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.242.67
--webserver_port=40551
--tserver_master_addrs=127.22.242.126:40059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.67
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:18.958390 4528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:18.960095 4528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:18.971315 4533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:18.971374 4534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:18.971887 4536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:18.972306 4528 server_base.cc:1034] running on GCE node
I20240503 14:14:18.972836 4528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:18.973678 4528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:18.974876 4528 hybrid_clock.cc:648] HybridClock initialized: now 1714745658974824 us; error 49 us; skew 500 ppm
I20240503 14:14:18.975173 4528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:18.977382 4528 webserver.cc:460] Webserver started at http://127.22.242.67:40551/ using document root <none> and password file <none>
I20240503 14:14:18.978082 4528 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:18.978217 4528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:18.983449 4528 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20240503 14:14:18.986291 4542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:18.987540 4528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:18.987710 4528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f105cb873c7b4f7a90ae1ee6088947c0"
format_stamp: "Formatted at 2024-05-03 14:14:10 on dist-test-slave-r3pd"
I20240503 14:14:18.988644 4528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:19.007736 4528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:19.008687 4528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:19.008973 4528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:19.009922 4528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:19.011914 4549 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240503 14:14:19.017457 4528 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 14:14:19.017575 4528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20240503 14:14:19.017757 4528 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 14:14:19.020736 4528 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 14:14:19.020845 4528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:19.021174 4549 tablet_bootstrap.cc:492] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0: Bootstrap starting.
I20240503 14:14:19.041220 4549 log.cc:824] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:19.045825 4549 tablet_bootstrap.cc:492] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:19.046505 4549 tablet_bootstrap.cc:492] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0: Bootstrap complete.
I20240503 14:14:19.048241 4549 ts_tablet_manager.cc:1367] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0: Time spent bootstrapping tablet: real 0.027s user 0.026s sys 0.001s
I20240503 14:14:19.059725 4549 raft_consensus.cc:348] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } }
I20240503 14:14:19.060878 4549 raft_consensus.cc:729] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f105cb873c7b4f7a90ae1ee6088947c0, State: Initialized, Role: FOLLOWER
I20240503 14:14:19.061614 4549 consensus_queue.cc:260] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } }
I20240503 14:14:19.061827 4549 raft_consensus.cc:388] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:19.061942 4549 raft_consensus.cc:482] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:19.062112 4549 raft_consensus.cc:3037] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:19.065753 4549 raft_consensus.cc:504] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } }
I20240503 14:14:19.066200 4549 leader_election.cc:302] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f105cb873c7b4f7a90ae1ee6088947c0; no voters:
I20240503 14:14:19.066885 4549 leader_election.cc:288] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240503 14:14:19.067054 4640 raft_consensus.cc:2781] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:19.068862 4549 ts_tablet_manager.cc:1398] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0: Time spent starting tablet: real 0.020s user 0.020s sys 0.000s
I20240503 14:14:19.069648 4640 raft_consensus.cc:686] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [term 2 LEADER]: Becoming Leader. State: Replica: f105cb873c7b4f7a90ae1ee6088947c0, State: Running, Role: LEADER
I20240503 14:14:19.070359 4640 consensus_queue.cc:237] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } }
I20240503 14:14:19.076480 4528 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.67:33985
I20240503 14:14:19.076608 4661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.67:33985 every 8 connection(s)
I20240503 14:14:19.077963 4528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:14:19.082454 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 4528
I20240503 14:14:19.083161 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.68:38509
--local_ip_for_outbound_sockets=127.22.242.68
--tserver_master_addrs=127.22.242.126:40059
--webserver_port=43177
--webserver_interface=127.22.242.68
--builtin_ntp_servers=127.22.242.84:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:14:19.089753 4662 heartbeater.cc:347] Connected to a master server at 127.22.242.126:40059
I20240503 14:14:19.090066 4662 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:19.090880 4662 heartbeater.cc:511] Master 127.22.242.126:40059 requested a full tablet report, sending...
I20240503 14:14:19.093668 3760 ts_manager.cc:194] Re-registered known tserver with Master: f105cb873c7b4f7a90ae1ee6088947c0 (127.22.242.67:33985)
I20240503 14:14:19.094204 3760 catalog_manager.cc:5484] T 80912b19b76b43e4b6a6fe7d2294ae06 P f105cb873c7b4f7a90ae1ee6088947c0 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:19.098636 3760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.67:34123
I20240503 14:14:19.101240 4662 heartbeater.cc:503] Master 127.22.242.126:40059 was elected leader, sending a full tablet report...
W20240503 14:14:19.415763 4666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:19.416105 4666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:19.416263 4666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:19.424101 4666 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:19.424278 4666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:19.424434 4666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.68
I20240503 14:14:19.433674 4666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.68:38509
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.242.68
--webserver_port=43177
--tserver_master_addrs=127.22.242.126:40059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.68
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:19.434998 4666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:19.436697 4666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:19.448408 4675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:19.448380 4677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:19.448432 4674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:19.449172 4666 server_base.cc:1034] running on GCE node
I20240503 14:14:19.449769 4666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:19.450681 4666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:19.451884 4666 hybrid_clock.cc:648] HybridClock initialized: now 1714745659451838 us; error 46 us; skew 500 ppm
I20240503 14:14:19.452229 4666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:19.454519 4666 webserver.cc:460] Webserver started at http://127.22.242.68:43177/ using document root <none> and password file <none>
I20240503 14:14:19.455247 4666 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:19.455385 4666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:19.460502 4666 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:19.463186 4683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:19.464574 4666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:19.464720 4666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3734b96f895b4d45b2ce758377826d43"
format_stamp: "Formatted at 2024-05-03 14:14:11 on dist-test-slave-r3pd"
I20240503 14:14:19.465497 4666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:19.485914 4666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:19.486773 4666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:19.487042 4666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:19.488011 4666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:19.489573 4666 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:19.489645 4666 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:19.489745 4666 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:19.489801 4666 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:19.521607 4666 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.68:38509
I20240503 14:14:19.521646 4795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.68:38509 every 8 connection(s)
I20240503 14:14:19.523033 4666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:14:19.525346 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 4666
I20240503 14:14:19.537487 4796 heartbeater.cc:347] Connected to a master server at 127.22.242.126:40059
I20240503 14:14:19.537863 4796 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:19.538893 4796 heartbeater.cc:511] Master 127.22.242.126:40059 requested a full tablet report, sending...
I20240503 14:14:19.540443 3760 ts_manager.cc:194] Re-registered known tserver with Master: 3734b96f895b4d45b2ce758377826d43 (127.22.242.68:38509)
I20240503 14:14:19.541226 3760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.68:57971
I20240503 14:14:20.016959 4731 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:20.027738 4011 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:20.028546 4591 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:20.036585 3880 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:20.203799 4011 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:20.204829 4731 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:20.205523 4591 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:20.214041 3880 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:20.392964 4731 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:20.394534 4011 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:20.402956 3880 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:20.403558 4591 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:20.469527 3901 consensus_queue.cc:237] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: true } }
I20240503 14:14:20.472023 4914 raft_consensus.cc:2932] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 3734b96f895b4d45b2ce758377826d43 (127.22.242.68) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: true } } }
I20240503 14:14:20.476861 3760 catalog_manager.cc:5484] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b reported cstate change: config changed from index -1 to 3, NON_VOTER 3734b96f895b4d45b2ce758377826d43 (127.22.242.68) added. New cstate: current_term: 1 leader_uuid: "fb22456acd9240fcbdbd570531f9a10b" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:14:20.483628 3835 consensus_peers.cc:485] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b -> Peer 3734b96f895b4d45b2ce758377826d43 (127.22.242.68:38509): Couldn't send request to peer 3734b96f895b4d45b2ce758377826d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0825740e9f794e25ab75b2b14e90ea96. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:20.491669 4032 consensus_queue.cc:237] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: true } }
I20240503 14:14:20.494141 4927 raft_consensus.cc:2932] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f105cb873c7b4f7a90ae1ee6088947c0 (127.22.242.67) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: true } } }
I20240503 14:14:20.501224 3760 catalog_manager.cc:5484] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 reported cstate change: config changed from index -1 to 2, NON_VOTER f105cb873c7b4f7a90ae1ee6088947c0 (127.22.242.67) added. New cstate: current_term: 1 leader_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:14:20.504051 3966 consensus_peers.cc:485] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 -> Peer f105cb873c7b4f7a90ae1ee6088947c0 (127.22.242.67:33985): Couldn't send request to peer f105cb873c7b4f7a90ae1ee6088947c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7885ec95faa045eeb65e4de68819b840. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:20.514088 3901 consensus_queue.cc:237] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: true } }
I20240503 14:14:20.515801 4913 raft_consensus.cc:2932] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3734b96f895b4d45b2ce758377826d43 (127.22.242.68) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: true } } }
W20240503 14:14:20.516289 3835 consensus_peers.cc:485] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b -> Peer 3734b96f895b4d45b2ce758377826d43 (127.22.242.68:38509): Couldn't send request to peer 3734b96f895b4d45b2ce758377826d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30216842696e4463a801892b43f5f77c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:20.520942 3757 catalog_manager.cc:5484] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b reported cstate change: config changed from index -1 to 2, NON_VOTER 3734b96f895b4d45b2ce758377826d43 (127.22.242.68) added. New cstate: current_term: 1 leader_uuid: "fb22456acd9240fcbdbd570531f9a10b" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:20.531737 4032 consensus_queue.cc:237] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: true } }
I20240503 14:14:20.533573 4927 raft_consensus.cc:2932] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 3734b96f895b4d45b2ce758377826d43 (127.22.242.68) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: true } } }
I20240503 14:14:20.537838 3757 catalog_manager.cc:5484] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 reported cstate change: config changed from index -1 to 3, NON_VOTER 3734b96f895b4d45b2ce758377826d43 (127.22.242.68) added. New cstate: current_term: 1 leader_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:14:20.538753 3966 consensus_peers.cc:485] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 -> Peer 3734b96f895b4d45b2ce758377826d43 (127.22.242.68:38509): Couldn't send request to peer 3734b96f895b4d45b2ce758377826d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0e57448a9404822aae4a81fa9cefba7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:20.544087 4796 heartbeater.cc:503] Master 127.22.242.126:40059 was elected leader, sending a full tablet report...
I20240503 14:14:20.550622 3901 consensus_queue.cc:237] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: true } }
I20240503 14:14:20.552191 4913 raft_consensus.cc:2932] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f105cb873c7b4f7a90ae1ee6088947c0 (127.22.242.67) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: true } } }
I20240503 14:14:20.556720 3760 catalog_manager.cc:5484] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b reported cstate change: config changed from index -1 to 2, NON_VOTER f105cb873c7b4f7a90ae1ee6088947c0 (127.22.242.67) added. New cstate: current_term: 1 leader_uuid: "fb22456acd9240fcbdbd570531f9a10b" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:14:20.558821 3835 consensus_peers.cc:485] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b -> Peer f105cb873c7b4f7a90ae1ee6088947c0 (127.22.242.67:33985): Couldn't send request to peer f105cb873c7b4f7a90ae1ee6088947c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5fc12cdadca4372b83ea547ca6daf4d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:20.923720 4997 ts_tablet_manager.cc:902] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43: Initiating tablet copy from peer fb22456acd9240fcbdbd570531f9a10b (127.22.242.65:36431)
I20240503 14:14:20.925694 4997 tablet_copy_client.cc:308] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:36431
I20240503 14:14:20.933188 3921 tablet_copy_service.cc:140] P fb22456acd9240fcbdbd570531f9a10b: Received BeginTabletCopySession request for tablet 0825740e9f794e25ab75b2b14e90ea96 from peer 3734b96f895b4d45b2ce758377826d43 ({username='slave'} at 127.22.242.68:37219)
I20240503 14:14:20.933442 3921 tablet_copy_service.cc:161] P fb22456acd9240fcbdbd570531f9a10b: Beginning new tablet copy session on tablet 0825740e9f794e25ab75b2b14e90ea96 from peer 3734b96f895b4d45b2ce758377826d43 at {username='slave'} at 127.22.242.68:37219: session id = 3734b96f895b4d45b2ce758377826d43-0825740e9f794e25ab75b2b14e90ea96
I20240503 14:14:20.936142 3921 tablet_copy_source_session.cc:205] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:20.940481 4997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0825740e9f794e25ab75b2b14e90ea96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:20.949864 4997 tablet_copy_client.cc:783] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:20.950191 4997 tablet_copy_client.cc:647] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:20.952360 4997 tablet_copy_client.cc:523] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:20.955185 4997 tablet_bootstrap.cc:492] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43: Bootstrap starting.
I20240503 14:14:20.961794 4997 log.cc:824] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:20.969923 4997 tablet_bootstrap.cc:492] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:20.970364 4997 tablet_bootstrap.cc:492] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43: Bootstrap complete.
I20240503 14:14:20.970790 4997 ts_tablet_manager.cc:1367] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.005s
I20240503 14:14:20.974807 4997 raft_consensus.cc:348] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: true } }
I20240503 14:14:20.975131 4997 raft_consensus.cc:729] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3734b96f895b4d45b2ce758377826d43, State: Initialized, Role: LEARNER
I20240503 14:14:20.975533 4997 consensus_queue.cc:260] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: true } }
I20240503 14:14:20.976792 4997 ts_tablet_manager.cc:1398] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240503 14:14:20.977705 3921 tablet_copy_service.cc:342] P fb22456acd9240fcbdbd570531f9a10b: Request end of tablet copy session 3734b96f895b4d45b2ce758377826d43-0825740e9f794e25ab75b2b14e90ea96 received from {username='slave'} at 127.22.242.68:37219
I20240503 14:14:20.977847 3921 tablet_copy_service.cc:433] P fb22456acd9240fcbdbd570531f9a10b: ending tablet copy session 3734b96f895b4d45b2ce758377826d43-0825740e9f794e25ab75b2b14e90ea96 on tablet 0825740e9f794e25ab75b2b14e90ea96 with peer 3734b96f895b4d45b2ce758377826d43
I20240503 14:14:21.004809 4997 ts_tablet_manager.cc:902] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43: Initiating tablet copy from peer fb22456acd9240fcbdbd570531f9a10b (127.22.242.65:36431)
I20240503 14:14:21.005960 4997 tablet_copy_client.cc:308] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:36431
I20240503 14:14:21.006707 3921 tablet_copy_service.cc:140] P fb22456acd9240fcbdbd570531f9a10b: Received BeginTabletCopySession request for tablet 30216842696e4463a801892b43f5f77c from peer 3734b96f895b4d45b2ce758377826d43 ({username='slave'} at 127.22.242.68:37219)
I20240503 14:14:21.006891 3921 tablet_copy_service.cc:161] P fb22456acd9240fcbdbd570531f9a10b: Beginning new tablet copy session on tablet 30216842696e4463a801892b43f5f77c from peer 3734b96f895b4d45b2ce758377826d43 at {username='slave'} at 127.22.242.68:37219: session id = 3734b96f895b4d45b2ce758377826d43-30216842696e4463a801892b43f5f77c
I20240503 14:14:21.009177 3921 tablet_copy_source_session.cc:205] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:21.010486 4997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30216842696e4463a801892b43f5f77c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:21.014752 4997 tablet_copy_client.cc:783] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:21.014989 4997 tablet_copy_client.cc:647] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:21.016877 4997 tablet_copy_client.cc:523] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:21.019255 4997 tablet_bootstrap.cc:492] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43: Bootstrap starting.
I20240503 14:14:21.023623 4997 tablet_bootstrap.cc:492] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:21.023993 4997 tablet_bootstrap.cc:492] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43: Bootstrap complete.
I20240503 14:14:21.024248 4997 ts_tablet_manager.cc:1367] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.002s
I20240503 14:14:21.024947 4997 raft_consensus.cc:348] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: true } }
I20240503 14:14:21.025131 4997 raft_consensus.cc:729] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3734b96f895b4d45b2ce758377826d43, State: Initialized, Role: LEARNER
I20240503 14:14:21.025318 4997 consensus_queue.cc:260] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: true } }
I20240503 14:14:21.025743 4997 ts_tablet_manager.cc:1398] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:14:21.026535 3921 tablet_copy_service.cc:342] P fb22456acd9240fcbdbd570531f9a10b: Request end of tablet copy session 3734b96f895b4d45b2ce758377826d43-30216842696e4463a801892b43f5f77c received from {username='slave'} at 127.22.242.68:37219
I20240503 14:14:21.026664 3921 tablet_copy_service.cc:433] P fb22456acd9240fcbdbd570531f9a10b: ending tablet copy session 3734b96f895b4d45b2ce758377826d43-30216842696e4463a801892b43f5f77c on tablet 30216842696e4463a801892b43f5f77c with peer 3734b96f895b4d45b2ce758377826d43
I20240503 14:14:21.036295 4997 ts_tablet_manager.cc:902] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43: Initiating tablet copy from peer 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66:46329)
I20240503 14:14:21.037405 4997 tablet_copy_client.cc:308] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:46329
I20240503 14:14:21.042924 4052 tablet_copy_service.cc:140] P 8f38339f4d434e39ba4cbc9cc0d236f0: Received BeginTabletCopySession request for tablet a0e57448a9404822aae4a81fa9cefba7 from peer 3734b96f895b4d45b2ce758377826d43 ({username='slave'} at 127.22.242.68:45091)
I20240503 14:14:21.043113 4052 tablet_copy_service.cc:161] P 8f38339f4d434e39ba4cbc9cc0d236f0: Beginning new tablet copy session on tablet a0e57448a9404822aae4a81fa9cefba7 from peer 3734b96f895b4d45b2ce758377826d43 at {username='slave'} at 127.22.242.68:45091: session id = 3734b96f895b4d45b2ce758377826d43-a0e57448a9404822aae4a81fa9cefba7
I20240503 14:14:21.045629 4052 tablet_copy_source_session.cc:205] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:21.047096 4997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e57448a9404822aae4a81fa9cefba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:21.051620 4997 tablet_copy_client.cc:783] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:21.051908 4997 tablet_copy_client.cc:647] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:21.053740 4997 tablet_copy_client.cc:523] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:21.056668 4997 tablet_bootstrap.cc:492] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43: Bootstrap starting.
I20240503 14:14:21.062595 4997 tablet_bootstrap.cc:492] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:21.062974 4997 tablet_bootstrap.cc:492] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43: Bootstrap complete.
I20240503 14:14:21.063205 4997 ts_tablet_manager.cc:1367] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:14:21.063997 4997 raft_consensus.cc:348] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: true } }
I20240503 14:14:21.064343 4997 raft_consensus.cc:729] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3734b96f895b4d45b2ce758377826d43, State: Initialized, Role: LEARNER
I20240503 14:14:21.064567 4997 consensus_queue.cc:260] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: true } }
I20240503 14:14:21.065079 4997 ts_tablet_manager.cc:1398] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 14:14:21.066097 4052 tablet_copy_service.cc:342] P 8f38339f4d434e39ba4cbc9cc0d236f0: Request end of tablet copy session 3734b96f895b4d45b2ce758377826d43-a0e57448a9404822aae4a81fa9cefba7 received from {username='slave'} at 127.22.242.68:45091
I20240503 14:14:21.066284 4052 tablet_copy_service.cc:433] P 8f38339f4d434e39ba4cbc9cc0d236f0: ending tablet copy session 3734b96f895b4d45b2ce758377826d43-a0e57448a9404822aae4a81fa9cefba7 on tablet a0e57448a9404822aae4a81fa9cefba7 with peer 3734b96f895b4d45b2ce758377826d43
I20240503 14:14:21.085253 5030 ts_tablet_manager.cc:902] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0: Initiating tablet copy from peer fb22456acd9240fcbdbd570531f9a10b (127.22.242.65:36431)
I20240503 14:14:21.087347 5030 tablet_copy_client.cc:308] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:36431
I20240503 14:14:21.094436 5040 ts_tablet_manager.cc:902] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0: Initiating tablet copy from peer 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66:46329)
I20240503 14:14:21.111562 3921 tablet_copy_service.cc:140] P fb22456acd9240fcbdbd570531f9a10b: Received BeginTabletCopySession request for tablet e5fc12cdadca4372b83ea547ca6daf4d from peer f105cb873c7b4f7a90ae1ee6088947c0 ({username='slave'} at 127.22.242.67:51613)
I20240503 14:14:21.111773 3921 tablet_copy_service.cc:161] P fb22456acd9240fcbdbd570531f9a10b: Beginning new tablet copy session on tablet e5fc12cdadca4372b83ea547ca6daf4d from peer f105cb873c7b4f7a90ae1ee6088947c0 at {username='slave'} at 127.22.242.67:51613: session id = f105cb873c7b4f7a90ae1ee6088947c0-e5fc12cdadca4372b83ea547ca6daf4d
I20240503 14:14:21.114073 5040 tablet_copy_client.cc:308] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:46329
I20240503 14:14:21.114270 3921 tablet_copy_source_session.cc:205] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:21.117179 5030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5fc12cdadca4372b83ea547ca6daf4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:21.121223 4052 tablet_copy_service.cc:140] P 8f38339f4d434e39ba4cbc9cc0d236f0: Received BeginTabletCopySession request for tablet 7885ec95faa045eeb65e4de68819b840 from peer f105cb873c7b4f7a90ae1ee6088947c0 ({username='slave'} at 127.22.242.67:53501)
I20240503 14:14:21.121414 4052 tablet_copy_service.cc:161] P 8f38339f4d434e39ba4cbc9cc0d236f0: Beginning new tablet copy session on tablet 7885ec95faa045eeb65e4de68819b840 from peer f105cb873c7b4f7a90ae1ee6088947c0 at {username='slave'} at 127.22.242.67:53501: session id = f105cb873c7b4f7a90ae1ee6088947c0-7885ec95faa045eeb65e4de68819b840
I20240503 14:14:21.123978 4052 tablet_copy_source_session.cc:205] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:21.124596 5030 tablet_copy_client.cc:783] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:21.124840 5030 tablet_copy_client.cc:647] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:21.126008 5040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7885ec95faa045eeb65e4de68819b840. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:21.127775 5030 tablet_copy_client.cc:523] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:21.132109 5030 tablet_bootstrap.cc:492] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0: Bootstrap starting.
I20240503 14:14:21.132172 5040 tablet_copy_client.cc:783] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:21.132490 5040 tablet_copy_client.cc:647] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:21.134202 5040 tablet_copy_client.cc:523] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:21.137504 5040 tablet_bootstrap.cc:492] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0: Bootstrap starting.
I20240503 14:14:21.139015 5030 tablet_bootstrap.cc:492] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:21.139482 5030 tablet_bootstrap.cc:492] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0: Bootstrap complete.
I20240503 14:14:21.139724 5030 ts_tablet_manager.cc:1367] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240503 14:14:21.140847 5030 raft_consensus.cc:348] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: true } }
I20240503 14:14:21.141112 5030 raft_consensus.cc:729] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f105cb873c7b4f7a90ae1ee6088947c0, State: Initialized, Role: LEARNER
I20240503 14:14:21.141444 5030 consensus_queue.cc:260] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: true } }
I20240503 14:14:21.142279 5030 ts_tablet_manager.cc:1398] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 14:14:21.143113 3921 tablet_copy_service.cc:342] P fb22456acd9240fcbdbd570531f9a10b: Request end of tablet copy session f105cb873c7b4f7a90ae1ee6088947c0-e5fc12cdadca4372b83ea547ca6daf4d received from {username='slave'} at 127.22.242.67:51613
I20240503 14:14:21.143252 3921 tablet_copy_service.cc:433] P fb22456acd9240fcbdbd570531f9a10b: ending tablet copy session f105cb873c7b4f7a90ae1ee6088947c0-e5fc12cdadca4372b83ea547ca6daf4d on tablet e5fc12cdadca4372b83ea547ca6daf4d with peer f105cb873c7b4f7a90ae1ee6088947c0
I20240503 14:14:21.143573 5040 tablet_bootstrap.cc:492] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:21.144461 5040 tablet_bootstrap.cc:492] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0: Bootstrap complete.
I20240503 14:14:21.144702 5040 ts_tablet_manager.cc:1367] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.002s
I20240503 14:14:21.145502 5040 raft_consensus.cc:348] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: true } }
I20240503 14:14:21.145660 5040 raft_consensus.cc:729] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f105cb873c7b4f7a90ae1ee6088947c0, State: Initialized, Role: LEARNER
I20240503 14:14:21.145845 5040 consensus_queue.cc:260] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: true } }
I20240503 14:14:21.146201 5040 ts_tablet_manager.cc:1398] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:14:21.146828 4052 tablet_copy_service.cc:342] P 8f38339f4d434e39ba4cbc9cc0d236f0: Request end of tablet copy session f105cb873c7b4f7a90ae1ee6088947c0-7885ec95faa045eeb65e4de68819b840 received from {username='slave'} at 127.22.242.67:53501
I20240503 14:14:21.146941 4052 tablet_copy_service.cc:433] P 8f38339f4d434e39ba4cbc9cc0d236f0: ending tablet copy session f105cb873c7b4f7a90ae1ee6088947c0-7885ec95faa045eeb65e4de68819b840 on tablet 7885ec95faa045eeb65e4de68819b840 with peer f105cb873c7b4f7a90ae1ee6088947c0
I20240503 14:14:21.392946 4751 raft_consensus.cc:1203] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:21.424453 4751 raft_consensus.cc:1203] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240503 14:14:21.468078 4751 raft_consensus.cc:1203] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240503 14:14:21.505803 4612 raft_consensus.cc:1203] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:21.698812 4612 raft_consensus.cc:1203] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:21.936908 5082 raft_consensus.cc:1050] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0: attempting to promote NON_VOTER 3734b96f895b4d45b2ce758377826d43 to VOTER
I20240503 14:14:21.937747 5082 consensus_queue.cc:237] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } }
I20240503 14:14:21.939656 4751 raft_consensus.cc:1261] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 [term 1 LEARNER]: Refusing update from remote peer 8f38339f4d434e39ba4cbc9cc0d236f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:21.940361 5083 consensus_queue.cc:1035] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:21.943747 5082 raft_consensus.cc:2932] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 3734b96f895b4d45b2ce758377826d43 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } } }
I20240503 14:14:21.944490 4750 raft_consensus.cc:2932] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 3734b96f895b4d45b2ce758377826d43 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } } }
I20240503 14:14:21.945073 5081 raft_consensus.cc:1050] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b: attempting to promote NON_VOTER 3734b96f895b4d45b2ce758377826d43 to VOTER
I20240503 14:14:21.945978 5081 consensus_queue.cc:237] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } }
I20240503 14:14:21.947897 4751 raft_consensus.cc:1261] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 [term 1 LEARNER]: Refusing update from remote peer fb22456acd9240fcbdbd570531f9a10b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:21.948612 5081 consensus_queue.cc:1035] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:21.949187 3757 catalog_manager.cc:5484] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 reported cstate change: config changed from index 3 to 4, 3734b96f895b4d45b2ce758377826d43 (127.22.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } } }
I20240503 14:14:21.952569 5080 raft_consensus.cc:2932] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3734b96f895b4d45b2ce758377826d43 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } } }
I20240503 14:14:21.953263 4751 raft_consensus.cc:2932] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3734b96f895b4d45b2ce758377826d43 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } } }
I20240503 14:14:21.956985 3760 catalog_manager.cc:5484] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b reported cstate change: config changed from index 2 to 3, 3734b96f895b4d45b2ce758377826d43 (127.22.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb22456acd9240fcbdbd570531f9a10b" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:22.018921 5080 raft_consensus.cc:1050] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b: attempting to promote NON_VOTER 3734b96f895b4d45b2ce758377826d43 to VOTER
I20240503 14:14:22.019528 5080 consensus_queue.cc:237] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } }
I20240503 14:14:22.021245 4751 raft_consensus.cc:1261] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 [term 1 LEARNER]: Refusing update from remote peer fb22456acd9240fcbdbd570531f9a10b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:22.021728 5080 consensus_queue.cc:1035] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:22.024291 4913 raft_consensus.cc:2932] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 3734b96f895b4d45b2ce758377826d43 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } } }
I20240503 14:14:22.024874 4751 raft_consensus.cc:2932] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 3734b96f895b4d45b2ce758377826d43 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } } }
I20240503 14:14:22.028430 3757 catalog_manager.cc:5484] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b reported cstate change: config changed from index 3 to 4, 3734b96f895b4d45b2ce758377826d43 (127.22.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb22456acd9240fcbdbd570531f9a10b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:22.051015 5080 raft_consensus.cc:1050] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b: attempting to promote NON_VOTER f105cb873c7b4f7a90ae1ee6088947c0 to VOTER
I20240503 14:14:22.051551 5080 consensus_queue.cc:237] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } }
I20240503 14:14:22.053411 4612 raft_consensus.cc:1261] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 LEARNER]: Refusing update from remote peer fb22456acd9240fcbdbd570531f9a10b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:22.053942 4913 consensus_queue.cc:1035] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:22.056476 5080 raft_consensus.cc:2932] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f105cb873c7b4f7a90ae1ee6088947c0 (127.22.242.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } } }
I20240503 14:14:22.057057 4612 raft_consensus.cc:2932] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f105cb873c7b4f7a90ae1ee6088947c0 (127.22.242.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } } }
I20240503 14:14:22.060859 3760 catalog_manager.cc:5484] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b reported cstate change: config changed from index 2 to 3, f105cb873c7b4f7a90ae1ee6088947c0 (127.22.242.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb22456acd9240fcbdbd570531f9a10b" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:22.097651 4032 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a0e57448a9404822aae4a81fa9cefba7"
dest_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42336
I20240503 14:14:22.097879 4032 raft_consensus.cc:595] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:22.115864 3901 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "30216842696e4463a801892b43f5f77c"
dest_uuid: "fb22456acd9240fcbdbd570531f9a10b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52474
I20240503 14:14:22.116137 3901 raft_consensus.cc:595] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:22.133630 3901 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5fc12cdadca4372b83ea547ca6daf4d"
dest_uuid: "fb22456acd9240fcbdbd570531f9a10b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52482
I20240503 14:14:22.133831 3901 raft_consensus.cc:595] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:22.151927 3901 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0825740e9f794e25ab75b2b14e90ea96"
dest_uuid: "fb22456acd9240fcbdbd570531f9a10b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52504
I20240503 14:14:22.152180 3901 raft_consensus.cc:595] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:22.202100 5083 raft_consensus.cc:1050] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0: attempting to promote NON_VOTER f105cb873c7b4f7a90ae1ee6088947c0 to VOTER
I20240503 14:14:22.202744 5083 consensus_queue.cc:237] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } }
I20240503 14:14:22.204636 4612 raft_consensus.cc:1261] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 LEARNER]: Refusing update from remote peer 8f38339f4d434e39ba4cbc9cc0d236f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:22.205159 5083 consensus_queue.cc:1035] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:22.207674 5082 raft_consensus.cc:2932] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f105cb873c7b4f7a90ae1ee6088947c0 (127.22.242.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } } }
I20240503 14:14:22.208307 4612 raft_consensus.cc:2932] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f105cb873c7b4f7a90ae1ee6088947c0 (127.22.242.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } } }
I20240503 14:14:22.212040 3757 catalog_manager.cc:5484] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 reported cstate change: config changed from index 2 to 3, f105cb873c7b4f7a90ae1ee6088947c0 (127.22.242.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:22.380879 4032 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a0e57448a9404822aae4a81fa9cefba7"
dest_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42364
I20240503 14:14:22.381070 4032 raft_consensus.cc:595] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:22.398749 3901 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "30216842696e4463a801892b43f5f77c"
dest_uuid: "fb22456acd9240fcbdbd570531f9a10b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52524
I20240503 14:14:22.398950 3901 raft_consensus.cc:595] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:22.417539 3901 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5fc12cdadca4372b83ea547ca6daf4d"
dest_uuid: "fb22456acd9240fcbdbd570531f9a10b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52550
I20240503 14:14:22.417802 3901 raft_consensus.cc:595] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:22.433251 4913 raft_consensus.cc:979] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b: : Instructing follower 3734b96f895b4d45b2ce758377826d43 to start an election
I20240503 14:14:22.433449 5081 raft_consensus.cc:1067] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Signalling peer 3734b96f895b4d45b2ce758377826d43 to start an election
I20240503 14:14:22.434285 4751 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "30216842696e4463a801892b43f5f77c"
dest_uuid: "3734b96f895b4d45b2ce758377826d43"
from {username='slave'} at 127.22.242.65:48017
I20240503 14:14:22.434556 4751 raft_consensus.cc:482] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:22.434757 4751 raft_consensus.cc:3037] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:22.435945 3901 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0825740e9f794e25ab75b2b14e90ea96"
dest_uuid: "fb22456acd9240fcbdbd570531f9a10b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52572
I20240503 14:14:22.436113 3901 raft_consensus.cc:595] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:22.438064 4751 raft_consensus.cc:504] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } }
I20240503 14:14:22.440209 4751 leader_election.cc:288] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 [CANDIDATE]: Term 2 election: Requested vote from peers fb22456acd9240fcbdbd570531f9a10b (127.22.242.65:36431)
I20240503 14:14:22.440831 3901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30216842696e4463a801892b43f5f77c" candidate_uuid: "3734b96f895b4d45b2ce758377826d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb22456acd9240fcbdbd570531f9a10b"
I20240503 14:14:22.441118 3901 raft_consensus.cc:3032] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:22.441202 3901 raft_consensus.cc:729] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb22456acd9240fcbdbd570531f9a10b, State: Running, Role: LEADER
I20240503 14:14:22.441516 3901 consensus_queue.cc:260] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } }
I20240503 14:14:22.441870 3901 raft_consensus.cc:3037] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:22.443781 3901 raft_consensus.cc:2454] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3734b96f895b4d45b2ce758377826d43 in term 2.
I20240503 14:14:22.444314 4684 leader_election.cc:302] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3734b96f895b4d45b2ce758377826d43, fb22456acd9240fcbdbd570531f9a10b; no voters:
I20240503 14:14:22.444731 5004 raft_consensus.cc:2781] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:22.445226 5004 raft_consensus.cc:686] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 [term 2 LEADER]: Becoming Leader. State: Replica: 3734b96f895b4d45b2ce758377826d43, State: Running, Role: LEADER
I20240503 14:14:22.445632 5004 consensus_queue.cc:237] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } }
I20240503 14:14:22.448577 3760 catalog_manager.cc:5484] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 reported cstate change: term changed from 1 to 2, leader changed from fb22456acd9240fcbdbd570531f9a10b (127.22.242.65) to 3734b96f895b4d45b2ce758377826d43 (127.22.242.68). New cstate: current_term: 2 leader_uuid: "3734b96f895b4d45b2ce758377826d43" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:14:22.460613 3745 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 30216842696e4463a801892b43f5f77c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 3734b96f895b4d45b2ce758377826d43 (127.22.242.68:38509) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240503 14:14:22.462513 4032 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7885ec95faa045eeb65e4de68819b840"
dest_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42378
I20240503 14:14:22.462706 4032 raft_consensus.cc:595] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:22.486726 5082 raft_consensus.cc:979] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0: : Instructing follower 3734b96f895b4d45b2ce758377826d43 to start an election
I20240503 14:14:22.486928 5082 raft_consensus.cc:1067] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Signalling peer 3734b96f895b4d45b2ce758377826d43 to start an election
I20240503 14:14:22.487699 4751 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a0e57448a9404822aae4a81fa9cefba7"
dest_uuid: "3734b96f895b4d45b2ce758377826d43"
from {username='slave'} at 127.22.242.66:49719
I20240503 14:14:22.487901 4751 raft_consensus.cc:482] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:22.488027 4751 raft_consensus.cc:3037] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:22.490074 4751 raft_consensus.cc:504] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } }
I20240503 14:14:22.490573 4751 leader_election.cc:288] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 [CANDIDATE]: Term 2 election: Requested vote from peers 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66:46329)
I20240503 14:14:22.491225 4032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e57448a9404822aae4a81fa9cefba7" candidate_uuid: "3734b96f895b4d45b2ce758377826d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0"
I20240503 14:14:22.491472 4032 raft_consensus.cc:3032] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:22.491568 4032 raft_consensus.cc:729] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f38339f4d434e39ba4cbc9cc0d236f0, State: Running, Role: LEADER
I20240503 14:14:22.491829 4032 consensus_queue.cc:260] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } }
I20240503 14:14:22.492133 4032 raft_consensus.cc:3037] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:22.494063 4032 raft_consensus.cc:2454] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3734b96f895b4d45b2ce758377826d43 in term 2.
I20240503 14:14:22.494514 4687 leader_election.cc:302] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3734b96f895b4d45b2ce758377826d43, 8f38339f4d434e39ba4cbc9cc0d236f0; no voters:
I20240503 14:14:22.494777 5004 raft_consensus.cc:2781] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:22.494938 5004 raft_consensus.cc:686] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 [term 2 LEADER]: Becoming Leader. State: Replica: 3734b96f895b4d45b2ce758377826d43, State: Running, Role: LEADER
I20240503 14:14:22.495231 5004 consensus_queue.cc:237] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } }
I20240503 14:14:22.497135 3760 catalog_manager.cc:5484] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 reported cstate change: term changed from 1 to 2, leader changed from 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66) to 3734b96f895b4d45b2ce758377826d43 (127.22.242.68). New cstate: current_term: 2 leader_uuid: "3734b96f895b4d45b2ce758377826d43" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:22.582990 4913 raft_consensus.cc:979] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b: : Instructing follower f105cb873c7b4f7a90ae1ee6088947c0 to start an election
I20240503 14:14:22.583233 4913 raft_consensus.cc:1067] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Signalling peer f105cb873c7b4f7a90ae1ee6088947c0 to start an election
I20240503 14:14:22.583931 4612 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e5fc12cdadca4372b83ea547ca6daf4d"
dest_uuid: "f105cb873c7b4f7a90ae1ee6088947c0"
from {username='slave'} at 127.22.242.65:52963
I20240503 14:14:22.584245 4612 raft_consensus.cc:482] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:22.584398 4612 raft_consensus.cc:3037] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:22.586269 4612 raft_consensus.cc:504] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } }
I20240503 14:14:22.587312 4612 leader_election.cc:288] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 [CANDIDATE]: Term 2 election: Requested vote from peers fb22456acd9240fcbdbd570531f9a10b (127.22.242.65:36431)
I20240503 14:14:22.587832 3901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5fc12cdadca4372b83ea547ca6daf4d" candidate_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb22456acd9240fcbdbd570531f9a10b"
I20240503 14:14:22.588058 3901 raft_consensus.cc:3032] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:22.588138 3901 raft_consensus.cc:729] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb22456acd9240fcbdbd570531f9a10b, State: Running, Role: LEADER
I20240503 14:14:22.588433 3901 consensus_queue.cc:260] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } }
I20240503 14:14:22.588693 3901 raft_consensus.cc:3037] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:22.590538 3901 raft_consensus.cc:2454] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f105cb873c7b4f7a90ae1ee6088947c0 in term 2.
I20240503 14:14:22.591048 4543 leader_election.cc:302] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: f105cb873c7b4f7a90ae1ee6088947c0, fb22456acd9240fcbdbd570531f9a10b; no voters:
I20240503 14:14:22.591356 5054 raft_consensus.cc:2781] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:22.591686 5054 raft_consensus.cc:686] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 [term 2 LEADER]: Becoming Leader. State: Replica: f105cb873c7b4f7a90ae1ee6088947c0, State: Running, Role: LEADER
I20240503 14:14:22.591972 5054 consensus_queue.cc:237] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } }
I20240503 14:14:22.594766 3760 catalog_manager.cc:5484] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 reported cstate change: term changed from 1 to 2, leader changed from fb22456acd9240fcbdbd570531f9a10b (127.22.242.65) to f105cb873c7b4f7a90ae1ee6088947c0 (127.22.242.67). New cstate: current_term: 2 leader_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:22.609875 5081 raft_consensus.cc:979] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b: : Instructing follower 3734b96f895b4d45b2ce758377826d43 to start an election
I20240503 14:14:22.610035 5081 raft_consensus.cc:1067] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Signalling peer 3734b96f895b4d45b2ce758377826d43 to start an election
I20240503 14:14:22.610694 4751 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0825740e9f794e25ab75b2b14e90ea96"
dest_uuid: "3734b96f895b4d45b2ce758377826d43"
from {username='slave'} at 127.22.242.65:48017
I20240503 14:14:22.610913 4751 raft_consensus.cc:482] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:22.611078 4751 raft_consensus.cc:3037] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:22.613171 4751 raft_consensus.cc:504] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } }
I20240503 14:14:22.613670 4751 leader_election.cc:288] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 [CANDIDATE]: Term 2 election: Requested vote from peers fb22456acd9240fcbdbd570531f9a10b (127.22.242.65:36431)
I20240503 14:14:22.614198 3901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0825740e9f794e25ab75b2b14e90ea96" candidate_uuid: "3734b96f895b4d45b2ce758377826d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fb22456acd9240fcbdbd570531f9a10b"
I20240503 14:14:22.614424 3901 raft_consensus.cc:3032] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:22.614501 3901 raft_consensus.cc:729] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb22456acd9240fcbdbd570531f9a10b, State: Running, Role: LEADER
I20240503 14:14:22.614773 3901 consensus_queue.cc:260] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } }
I20240503 14:14:22.615087 3901 raft_consensus.cc:3037] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:22.616958 3901 raft_consensus.cc:2454] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3734b96f895b4d45b2ce758377826d43 in term 2.
I20240503 14:14:22.617317 4684 leader_election.cc:302] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3734b96f895b4d45b2ce758377826d43, fb22456acd9240fcbdbd570531f9a10b; no voters:
I20240503 14:14:22.617648 5004 raft_consensus.cc:2781] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:22.617779 5004 raft_consensus.cc:686] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 [term 2 LEADER]: Becoming Leader. State: Replica: 3734b96f895b4d45b2ce758377826d43, State: Running, Role: LEADER
I20240503 14:14:22.618062 5004 consensus_queue.cc:237] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } }
I20240503 14:14:22.619972 3760 catalog_manager.cc:5484] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 reported cstate change: term changed from 1 to 2, leader changed from fb22456acd9240fcbdbd570531f9a10b (127.22.242.65) to 3734b96f895b4d45b2ce758377826d43 (127.22.242.68). New cstate: current_term: 2 leader_uuid: "3734b96f895b4d45b2ce758377826d43" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:22.719218 4032 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7885ec95faa045eeb65e4de68819b840"
dest_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42408
I20240503 14:14:22.719417 4032 raft_consensus.cc:595] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:22.747819 5083 raft_consensus.cc:979] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0: : Instructing follower f105cb873c7b4f7a90ae1ee6088947c0 to start an election
I20240503 14:14:22.748060 5083 raft_consensus.cc:1067] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Signalling peer f105cb873c7b4f7a90ae1ee6088947c0 to start an election
I20240503 14:14:22.748689 4612 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7885ec95faa045eeb65e4de68819b840"
dest_uuid: "f105cb873c7b4f7a90ae1ee6088947c0"
from {username='slave'} at 127.22.242.66:46199
I20240503 14:14:22.748900 4612 raft_consensus.cc:482] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:22.749002 4612 raft_consensus.cc:3037] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:22.750854 4612 raft_consensus.cc:504] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } }
I20240503 14:14:22.751420 4612 leader_election.cc:288] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 [CANDIDATE]: Term 2 election: Requested vote from peers 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66:46329)
I20240503 14:14:22.751941 4032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7885ec95faa045eeb65e4de68819b840" candidate_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0"
I20240503 14:14:22.752099 4032 raft_consensus.cc:3032] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:22.752229 4032 raft_consensus.cc:729] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f38339f4d434e39ba4cbc9cc0d236f0, State: Running, Role: LEADER
I20240503 14:14:22.752440 4032 consensus_queue.cc:260] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } }
I20240503 14:14:22.752682 4032 raft_consensus.cc:3037] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:22.754421 4032 raft_consensus.cc:2454] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f105cb873c7b4f7a90ae1ee6088947c0 in term 2.
I20240503 14:14:22.754830 4546 leader_election.cc:302] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8f38339f4d434e39ba4cbc9cc0d236f0, f105cb873c7b4f7a90ae1ee6088947c0; no voters:
I20240503 14:14:22.755141 5054 raft_consensus.cc:2781] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:22.755292 5054 raft_consensus.cc:686] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 [term 2 LEADER]: Becoming Leader. State: Replica: f105cb873c7b4f7a90ae1ee6088947c0, State: Running, Role: LEADER
I20240503 14:14:22.755537 5054 consensus_queue.cc:237] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } }
I20240503 14:14:22.757432 3760 catalog_manager.cc:5484] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 reported cstate change: term changed from 1 to 2, leader changed from 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66) to f105cb873c7b4f7a90ae1ee6088947c0 (127.22.242.67). New cstate: current_term: 2 leader_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:22.952773 4032 raft_consensus.cc:1261] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 2 FOLLOWER]: Refusing update from remote peer 3734b96f895b4d45b2ce758377826d43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:14:22.953593 5004 consensus_queue.cc:1035] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:23.015455 3901 raft_consensus.cc:1261] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 2 FOLLOWER]: Refusing update from remote peer 3734b96f895b4d45b2ce758377826d43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:14:23.016250 5004 consensus_queue.cc:1035] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:23.039297 3901 raft_consensus.cc:1261] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 2 FOLLOWER]: Refusing update from remote peer 3734b96f895b4d45b2ce758377826d43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:14:23.039832 5306 consensus_queue.cc:1035] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:23.091041 4751 consensus_queue.cc:237] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } }
I20240503 14:14:23.091954 5238 raft_consensus.cc:2932] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER fb22456acd9240fcbdbd570531f9a10b (127.22.242.65) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } } }
I20240503 14:14:23.094944 3745 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 30216842696e4463a801892b43f5f77c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 14:14:23.096118 3757 catalog_manager.cc:5484] T 30216842696e4463a801892b43f5f77c P 3734b96f895b4d45b2ce758377826d43 reported cstate change: config changed from index 3 to 5, VOTER fb22456acd9240fcbdbd570531f9a10b (127.22.242.65) evicted. New cstate: current_term: 2 leader_uuid: "3734b96f895b4d45b2ce758377826d43" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:23.099606 3880 tablet_service.cc:1514] Processing DeleteTablet for tablet 30216842696e4463a801892b43f5f77c with delete_type TABLET_DATA_TOMBSTONED (TS fb22456acd9240fcbdbd570531f9a10b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52236
I20240503 14:14:23.100373 5309 tablet_replica.cc:330] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b: stopping tablet replica
I20240503 14:14:23.100734 5309 raft_consensus.cc:2229] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:23.100937 5309 raft_consensus.cc:2258] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:23.102320 5309 ts_tablet_manager.cc:1875] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:23.106963 5309 ts_tablet_manager.cc:1888] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 14:14:23.107085 5309 log.cc:1196] T 30216842696e4463a801892b43f5f77c P fb22456acd9240fcbdbd570531f9a10b: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/30216842696e4463a801892b43f5f77c
I20240503 14:14:23.107944 3744 catalog_manager.cc:4830] TS fb22456acd9240fcbdbd570531f9a10b (127.22.242.65:36431): tablet 30216842696e4463a801892b43f5f77c (table rebalance_test_table_1 [id=0615440c9e1b4b49a0cfbc6d828ef5f8]) successfully deleted
I20240503 14:14:23.137014 4751 consensus_queue.cc:237] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } }
I20240503 14:14:23.137925 5238 raft_consensus.cc:2932] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } } }
I20240503 14:14:23.140515 3745 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a0e57448a9404822aae4a81fa9cefba7 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 14:14:23.142007 3757 catalog_manager.cc:5484] T a0e57448a9404822aae4a81fa9cefba7 P 3734b96f895b4d45b2ce758377826d43 reported cstate change: config changed from index 4 to 6, VOTER 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66) evicted. New cstate: current_term: 2 leader_uuid: "3734b96f895b4d45b2ce758377826d43" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:23.145299 4011 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e57448a9404822aae4a81fa9cefba7 with delete_type TABLET_DATA_TOMBSTONED (TS 8f38339f4d434e39ba4cbc9cc0d236f0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42158
I20240503 14:14:23.146150 5310 tablet_replica.cc:330] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0: stopping tablet replica
I20240503 14:14:23.146510 5310 raft_consensus.cc:2229] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:23.146723 5310 raft_consensus.cc:2258] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:23.147925 5310 ts_tablet_manager.cc:1875] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:23.152558 5310 ts_tablet_manager.cc:1888] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:14:23.152680 5310 log.cc:1196] T a0e57448a9404822aae4a81fa9cefba7 P 8f38339f4d434e39ba4cbc9cc0d236f0: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/a0e57448a9404822aae4a81fa9cefba7
I20240503 14:14:23.153417 3747 catalog_manager.cc:4830] TS 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66:46329): tablet a0e57448a9404822aae4a81fa9cefba7 (table rebalance_test_table_3 [id=d8bc56b41046428cbe11ffc734ef8e75]) successfully deleted
I20240503 14:14:23.213677 3901 raft_consensus.cc:1261] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 2 FOLLOWER]: Refusing update from remote peer f105cb873c7b4f7a90ae1ee6088947c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:14:23.214345 5054 consensus_queue.cc:1035] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb22456acd9240fcbdbd570531f9a10b" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36431 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:23.255925 4612 consensus_queue.cc:237] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } }
I20240503 14:14:23.256858 5054 raft_consensus.cc:2932] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER fb22456acd9240fcbdbd570531f9a10b (127.22.242.65) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } } }
I20240503 14:14:23.259707 3745 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e5fc12cdadca4372b83ea547ca6daf4d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 14:14:23.261127 3757 catalog_manager.cc:5484] T e5fc12cdadca4372b83ea547ca6daf4d P f105cb873c7b4f7a90ae1ee6088947c0 reported cstate change: config changed from index 3 to 5, VOTER fb22456acd9240fcbdbd570531f9a10b (127.22.242.65) evicted. New cstate: current_term: 2 leader_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:23.264529 3880 tablet_service.cc:1514] Processing DeleteTablet for tablet e5fc12cdadca4372b83ea547ca6daf4d with delete_type TABLET_DATA_TOMBSTONED (TS fb22456acd9240fcbdbd570531f9a10b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52236
I20240503 14:14:23.264854 5309 tablet_replica.cc:330] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b: stopping tablet replica
I20240503 14:14:23.265033 5309 raft_consensus.cc:2229] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:23.265211 5309 raft_consensus.cc:2258] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:23.265879 5309 ts_tablet_manager.cc:1875] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:23.270114 5309 ts_tablet_manager.cc:1888] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 14:14:23.270226 5309 log.cc:1196] T e5fc12cdadca4372b83ea547ca6daf4d P fb22456acd9240fcbdbd570531f9a10b: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/e5fc12cdadca4372b83ea547ca6daf4d
I20240503 14:14:23.270841 3744 catalog_manager.cc:4830] TS fb22456acd9240fcbdbd570531f9a10b (127.22.242.65:36431): tablet e5fc12cdadca4372b83ea547ca6daf4d (table rebalance_test_table_0 [id=42b5ab15f2f048179c8e0c1a6c99d22f]) successfully deleted
I20240503 14:14:23.287425 4751 consensus_queue.cc:237] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } }
I20240503 14:14:23.288275 5306 raft_consensus.cc:2932] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER fb22456acd9240fcbdbd570531f9a10b (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } } }
I20240503 14:14:23.290966 3745 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0825740e9f794e25ab75b2b14e90ea96 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 14:14:23.292271 3757 catalog_manager.cc:5484] T 0825740e9f794e25ab75b2b14e90ea96 P 3734b96f895b4d45b2ce758377826d43 reported cstate change: config changed from index 4 to 6, VOTER fb22456acd9240fcbdbd570531f9a10b (127.22.242.65) evicted. New cstate: current_term: 2 leader_uuid: "3734b96f895b4d45b2ce758377826d43" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3734b96f895b4d45b2ce758377826d43" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 38509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:23.295325 3880 tablet_service.cc:1514] Processing DeleteTablet for tablet 0825740e9f794e25ab75b2b14e90ea96 with delete_type TABLET_DATA_TOMBSTONED (TS fb22456acd9240fcbdbd570531f9a10b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52236
I20240503 14:14:23.295650 5309 tablet_replica.cc:330] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b: stopping tablet replica
I20240503 14:14:23.295833 5309 raft_consensus.cc:2229] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:23.296017 5309 raft_consensus.cc:2258] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:23.296703 5309 ts_tablet_manager.cc:1875] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:23.300801 5309 ts_tablet_manager.cc:1888] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:14:23.300931 5309 log.cc:1196] T 0825740e9f794e25ab75b2b14e90ea96 P fb22456acd9240fcbdbd570531f9a10b: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/0825740e9f794e25ab75b2b14e90ea96
I20240503 14:14:23.301633 3744 catalog_manager.cc:4830] TS fb22456acd9240fcbdbd570531f9a10b (127.22.242.65:36431): tablet 0825740e9f794e25ab75b2b14e90ea96 (table rebalance_test_table_2 [id=4cecb121529244ab8fc498a140f28970]) successfully deleted
I20240503 14:14:23.327291 4032 raft_consensus.cc:1261] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 2 FOLLOWER]: Refusing update from remote peer f105cb873c7b4f7a90ae1ee6088947c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:14:23.327876 5054 consensus_queue.cc:1035] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f38339f4d434e39ba4cbc9cc0d236f0" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 46329 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:23.397890 4612 consensus_queue.cc:237] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } }
I20240503 14:14:23.398805 5246 raft_consensus.cc:2932] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } } }
I20240503 14:14:23.401481 3745 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7885ec95faa045eeb65e4de68819b840 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 14:14:23.402863 3760 catalog_manager.cc:5484] T 7885ec95faa045eeb65e4de68819b840 P f105cb873c7b4f7a90ae1ee6088947c0 reported cstate change: config changed from index 3 to 5, VOTER 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66) evicted. New cstate: current_term: 2 leader_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f105cb873c7b4f7a90ae1ee6088947c0" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 33985 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:23.406193 4011 tablet_service.cc:1514] Processing DeleteTablet for tablet 7885ec95faa045eeb65e4de68819b840 with delete_type TABLET_DATA_TOMBSTONED (TS 8f38339f4d434e39ba4cbc9cc0d236f0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42158
I20240503 14:14:23.406520 5310 tablet_replica.cc:330] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0: stopping tablet replica
I20240503 14:14:23.406689 5310 raft_consensus.cc:2229] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:23.406844 5310 raft_consensus.cc:2258] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:23.407615 5310 ts_tablet_manager.cc:1875] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:23.411782 5310 ts_tablet_manager.cc:1888] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 14:14:23.411886 5310 log.cc:1196] T 7885ec95faa045eeb65e4de68819b840 P 8f38339f4d434e39ba4cbc9cc0d236f0: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/7885ec95faa045eeb65e4de68819b840
I20240503 14:14:23.412591 3747 catalog_manager.cc:4830] TS 8f38339f4d434e39ba4cbc9cc0d236f0 (127.22.242.66:46329): tablet 7885ec95faa045eeb65e4de68819b840 (table rebalance_test_table_1 [id=0615440c9e1b4b49a0cfbc6d828ef5f8]) successfully deleted
I20240503 14:14:23.582830 4731 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:23.585225 4591 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:23.595242 4011 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:14:23.604364 3880 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:23.747743 4731 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:23.748570 4011 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:14:23.757383 3880 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:23.761653 4591 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:24.031577 4731 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:24.033454 4591 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:24.034746 4011 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:14:24.040423 3880 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
606ba1dfda514492a9c4398a3acebca4 | 127.22.242.126:40059 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.242.84:43371 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
3734b96f895b4d45b2ce758377826d43 | 127.22.242.68:38509 | HEALTHY | <none> | 3 | 0
8f38339f4d434e39ba4cbc9cc0d236f0 | 127.22.242.66:46329 | HEALTHY | <none> | 4 | 0
f105cb873c7b4f7a90ae1ee6088947c0 | 127.22.242.67:33985 | HEALTHY | <none> | 3 | 0
fb22456acd9240fcbdbd570531f9a10b | 127.22.242.65:36431 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.22.242.65 | experimental | 127.22.242.65:36431
local_ip_for_outbound_sockets | 127.22.242.66 | experimental | 127.22.242.66:46329
local_ip_for_outbound_sockets | 127.22.242.67 | experimental | 127.22.242.67:33985
local_ip_for_outbound_sockets | 127.22.242.68 | experimental | 127.22.242.68:38509
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.242.65:36431
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.242.66:46329
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.242.67:33985
server_dump_info_path | /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.22.242.68:38509
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.242.84:43371 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:14:24.100658 23497 log_verifier.cc:126] Checking tablet 0825740e9f794e25ab75b2b14e90ea96
I20240503 14:14:24.109447 23497 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 0825740e9f794e25ab75b2b14e90ea96
I20240503 14:14:24.109572 23497 log_verifier.cc:126] Checking tablet 1798538a1d0e46b7b8f95d275afe4662
I20240503 14:14:24.117086 23497 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 1798538a1d0e46b7b8f95d275afe4662
I20240503 14:14:24.117193 23497 log_verifier.cc:126] Checking tablet 2a3ea6cfa9714f89ba973d355e74ffab
I20240503 14:14:24.124511 23497 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 2a3ea6cfa9714f89ba973d355e74ffab
I20240503 14:14:24.124604 23497 log_verifier.cc:126] Checking tablet 30216842696e4463a801892b43f5f77c
I20240503 14:14:24.132647 23497 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 30216842696e4463a801892b43f5f77c
I20240503 14:14:24.132752 23497 log_verifier.cc:126] Checking tablet 3b1f3f7a061b40918d9666f19e3ccfdb
I20240503 14:14:24.139781 23497 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 3b1f3f7a061b40918d9666f19e3ccfdb
I20240503 14:14:24.139873 23497 log_verifier.cc:126] Checking tablet 7885ec95faa045eeb65e4de68819b840
I20240503 14:14:24.147603 23497 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 7885ec95faa045eeb65e4de68819b840
I20240503 14:14:24.147702 23497 log_verifier.cc:126] Checking tablet 80912b19b76b43e4b6a6fe7d2294ae06
I20240503 14:14:24.155159 23497 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 80912b19b76b43e4b6a6fe7d2294ae06
I20240503 14:14:24.155251 23497 log_verifier.cc:126] Checking tablet 816ac90d991a4a7b9f1af35b95848900
I20240503 14:14:24.162467 23497 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 816ac90d991a4a7b9f1af35b95848900
I20240503 14:14:24.162555 23497 log_verifier.cc:126] Checking tablet a0e57448a9404822aae4a81fa9cefba7
I20240503 14:14:24.170692 23497 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a0e57448a9404822aae4a81fa9cefba7
I20240503 14:14:24.170789 23497 log_verifier.cc:126] Checking tablet b9d129cb330a4c1e9341798b08b55094
I20240503 14:14:24.212934 23497 log_verifier.cc:177] Verified matching terms for 1 ops in tablet b9d129cb330a4c1e9341798b08b55094
I20240503 14:14:24.213016 23497 log_verifier.cc:126] Checking tablet baad63a6a5b64c3c952900c382c9f809
I20240503 14:14:24.220254 23497 log_verifier.cc:177] Verified matching terms for 1 ops in tablet baad63a6a5b64c3c952900c382c9f809
I20240503 14:14:24.220335 23497 log_verifier.cc:126] Checking tablet e5fc12cdadca4372b83ea547ca6daf4d
I20240503 14:14:24.228363 23497 log_verifier.cc:177] Verified matching terms for 5 ops in tablet e5fc12cdadca4372b83ea547ca6daf4d
I20240503 14:14:24.228446 23497 log_verifier.cc:126] Checking tablet ec9f52d1b5744db68351657a42a78a5b
I20240503 14:14:24.235769 23497 log_verifier.cc:177] Verified matching terms for 2 ops in tablet ec9f52d1b5744db68351657a42a78a5b
I20240503 14:14:24.236555 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 3819
I20240503 14:14:24.237964 3940 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:24.352136 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 3819
I20240503 14:14:24.380973 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 3950
I20240503 14:14:24.382055 4071 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:24.492240 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 3950
I20240503 14:14:24.513798 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 4528
I20240503 14:14:24.519762 4655 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:24.627666 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 4528
I20240503 14:14:24.647375 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 4666
I20240503 14:14:24.653107 4790 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:24.756316 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 4666
I20240503 14:14:24.775084 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 3729
I20240503 14:14:24.776136 3790 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:24.899379 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 3729
2024-05-03T14:14:24Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1 (16076 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (16076 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3
I20240503 14:14:24.957226 23497 test_util.cc:268] Using random seed: 1446865680
I20240503 14:14:24.958902 23497 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:14:24.958971 23497 ts_itest-base.cc:113] --------------
I20240503 14:14:24.959018 23497 ts_itest-base.cc:114] 6 tablet servers
I20240503 14:14:24.959082 23497 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:14:24.959132 23497 ts_itest-base.cc:116] --------------
2024-05-03T14:14:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:14:24Z Disabled control of system clock
I20240503 14:14:25.005218 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.242.126:41785
--webserver_interface=127.22.242.126
--webserver_port=0
--builtin_ntp_servers=127.22.242.84:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.242.126:41785
--location_mapping_cmd=/tmp/dist-test-taskqfBHnl/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 14:14:25.334784 5470 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 14:14:25.335134 5470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:25.335245 5470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:25.335342 5470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:25.342762 5470 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:25.342875 5470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:14:25.342994 5470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:25.343058 5470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:14:25.343151 5470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:14:25.352095 5470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskqfBHnl/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--ipki_ca_key_size=768
--master_addresses=127.22.242.126:41785
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.242.126:41785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.242.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:25.353413 5470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:25.355104 5470 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:25.365306 5476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:25.365345 5478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:25.365381 5475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:25.366142 5470 server_base.cc:1034] running on GCE node
I20240503 14:14:25.366855 5470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:25.368155 5470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:25.369401 5470 hybrid_clock.cc:648] HybridClock initialized: now 1714745665369338 us; error 39 us; skew 500 ppm
I20240503 14:14:25.369700 5470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:25.371785 5470 webserver.cc:460] Webserver started at http://127.22.242.126:45721/ using document root <none> and password file <none>
I20240503 14:14:25.372509 5470 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:25.372627 5470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:25.372905 5470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:25.375349 5470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1ed7e1aee52c4fb7a43d19399a377bfb"
format_stamp: "Formatted at 2024-05-03 14:14:25 on dist-test-slave-r3pd"
I20240503 14:14:25.376066 5470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1ed7e1aee52c4fb7a43d19399a377bfb"
format_stamp: "Formatted at 2024-05-03 14:14:25 on dist-test-slave-r3pd"
I20240503 14:14:25.380903 5470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240503 14:14:25.383963 5484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:25.385248 5470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 14:14:25.385399 5470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1ed7e1aee52c4fb7a43d19399a377bfb"
format_stamp: "Formatted at 2024-05-03 14:14:25 on dist-test-slave-r3pd"
I20240503 14:14:25.385568 5470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:25.405452 5470 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:25.406317 5470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:25.406607 5470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:25.425060 5470 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.126:41785
I20240503 14:14:25.425081 5535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.126:41785 every 8 connection(s)
I20240503 14:14:25.426431 5470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:14:25.430123 5536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:25.433962 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 5470
I20240503 14:14:25.434198 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:14:25.439879 5536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb: Bootstrap starting.
I20240503 14:14:25.442826 5536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:25.443967 5536 log.cc:824] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:25.446803 5536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb: No bootstrap required, opened a new log
I20240503 14:14:25.452582 5536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed7e1aee52c4fb7a43d19399a377bfb" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 41785 } }
I20240503 14:14:25.452893 5536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:25.452980 5536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed7e1aee52c4fb7a43d19399a377bfb, State: Initialized, Role: FOLLOWER
I20240503 14:14:25.453510 5536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed7e1aee52c4fb7a43d19399a377bfb" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 41785 } }
I20240503 14:14:25.453701 5536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:25.453819 5536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:25.453963 5536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:25.455900 5536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed7e1aee52c4fb7a43d19399a377bfb" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 41785 } }
I20240503 14:14:25.456377 5536 leader_election.cc:302] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ed7e1aee52c4fb7a43d19399a377bfb; no voters:
I20240503 14:14:25.456930 5536 leader_election.cc:288] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:14:25.457054 5541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:25.457672 5541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb [term 1 LEADER]: Becoming Leader. State: Replica: 1ed7e1aee52c4fb7a43d19399a377bfb, State: Running, Role: LEADER
I20240503 14:14:25.458200 5541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed7e1aee52c4fb7a43d19399a377bfb" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 41785 } }
I20240503 14:14:25.458824 5536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:14:25.461119 5543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ed7e1aee52c4fb7a43d19399a377bfb. Latest consensus state: current_term: 1 leader_uuid: "1ed7e1aee52c4fb7a43d19399a377bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed7e1aee52c4fb7a43d19399a377bfb" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 41785 } } }
I20240503 14:14:25.461493 5543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb [sys.catalog]: This master's current role is: LEADER
I20240503 14:14:25.461359 5542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ed7e1aee52c4fb7a43d19399a377bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed7e1aee52c4fb7a43d19399a377bfb" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 41785 } } }
I20240503 14:14:25.461887 5542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb [sys.catalog]: This master's current role is: LEADER
I20240503 14:14:25.462996 5549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:14:25.467612 5549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:14:25.475292 5549 catalog_manager.cc:1347] Generated new cluster ID: 8d328c907ff643ffaaee4ac873b2a7a0
I20240503 14:14:25.475399 5549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:14:25.498535 5549 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:14:25.499737 5549 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:14:25.511672 5549 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1ed7e1aee52c4fb7a43d19399a377bfb: Generated new TSK 0
I20240503 14:14:25.512573 5549 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:14:25.518613 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.65:0
--local_ip_for_outbound_sockets=127.22.242.65
--webserver_interface=127.22.242.65
--webserver_port=0
--tserver_master_addrs=127.22.242.126:41785
--builtin_ntp_servers=127.22.242.84:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:14:25.855136 5560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:25.855427 5560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:25.855549 5560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:25.863209 5560 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:25.863330 5560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:25.863498 5560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.65
I20240503 14:14:25.872510 5560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.242.65
--webserver_port=0
--tserver_master_addrs=127.22.242.126:41785
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.65
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:25.873750 5560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:25.875454 5560 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:25.887434 5566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:25.887456 5565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:25.887660 5568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:25.888787 5560 server_base.cc:1034] running on GCE node
I20240503 14:14:25.889382 5560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:25.890336 5560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:25.891532 5560 hybrid_clock.cc:648] HybridClock initialized: now 1714745665891490 us; error 26 us; skew 500 ppm
I20240503 14:14:25.891853 5560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:25.894135 5560 webserver.cc:460] Webserver started at http://127.22.242.65:38035/ using document root <none> and password file <none>
I20240503 14:14:25.894836 5560 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:25.894953 5560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:25.895269 5560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:25.897810 5560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33"
format_stamp: "Formatted at 2024-05-03 14:14:25 on dist-test-slave-r3pd"
I20240503 14:14:25.898566 5560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33"
format_stamp: "Formatted at 2024-05-03 14:14:25 on dist-test-slave-r3pd"
I20240503 14:14:25.903676 5560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:14:25.907171 5574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:25.908562 5560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 14:14:25.908721 5560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33"
format_stamp: "Formatted at 2024-05-03 14:14:25 on dist-test-slave-r3pd"
I20240503 14:14:25.908924 5560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:25.925238 5560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:25.926110 5560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:25.926380 5560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:25.927287 5560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:25.928783 5560 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:25.928884 5560 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:25.929001 5560 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:25.929075 5560 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:25.960484 5560 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.65:36157
I20240503 14:14:25.960542 5686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.65:36157 every 8 connection(s)
I20240503 14:14:25.961791 5560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:14:25.971453 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 5560
I20240503 14:14:25.971676 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:14:25.974573 5687 heartbeater.cc:347] Connected to a master server at 127.22.242.126:41785
I20240503 14:14:25.974853 5687 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:25.975190 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.66:0
--local_ip_for_outbound_sockets=127.22.242.66
--webserver_interface=127.22.242.66
--webserver_port=0
--tserver_master_addrs=127.22.242.126:41785
--builtin_ntp_servers=127.22.242.84:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:14:25.975673 5687 heartbeater.cc:511] Master 127.22.242.126:41785 requested a full tablet report, sending...
I20240503 14:14:26.043812 5500 ts_manager.cc:194] Registered new tserver with Master: e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157)
I20240503 14:14:26.045564 5500 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.65:59851
W20240503 14:14:26.329944 5692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:26.330268 5692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:26.330404 5692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:26.337992 5692 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:26.338140 5692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:26.338303 5692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.66
I20240503 14:14:26.347097 5692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.242.66
--webserver_port=0
--tserver_master_addrs=127.22.242.126:41785
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.66
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:26.348435 5692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:26.350092 5692 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:26.361447 5697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:26.361505 5700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:26.361506 5698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:26.362461 5692 server_base.cc:1034] running on GCE node
I20240503 14:14:26.363126 5692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:26.363950 5692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:26.365149 5692 hybrid_clock.cc:648] HybridClock initialized: now 1714745666365054 us; error 87 us; skew 500 ppm
I20240503 14:14:26.365446 5692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:26.371552 5692 webserver.cc:460] Webserver started at http://127.22.242.66:43509/ using document root <none> and password file <none>
I20240503 14:14:26.372339 5692 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:26.372462 5692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:26.372767 5692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:26.375305 5692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "5361a07e89ad47298aabe717a4f2867f"
format_stamp: "Formatted at 2024-05-03 14:14:26 on dist-test-slave-r3pd"
I20240503 14:14:26.376037 5692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "5361a07e89ad47298aabe717a4f2867f"
format_stamp: "Formatted at 2024-05-03 14:14:26 on dist-test-slave-r3pd"
I20240503 14:14:26.381039 5692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20240503 14:14:26.384347 5706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:26.385601 5692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 14:14:26.385756 5692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "5361a07e89ad47298aabe717a4f2867f"
format_stamp: "Formatted at 2024-05-03 14:14:26 on dist-test-slave-r3pd"
I20240503 14:14:26.385957 5692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:26.412649 5692 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:26.413594 5692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:26.413890 5692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:26.414850 5692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:26.416401 5692 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:26.416501 5692 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:26.416702 5692 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:26.416828 5692 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:26.451344 5692 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.66:41071
I20240503 14:14:26.451409 5818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.66:41071 every 8 connection(s)
I20240503 14:14:26.452636 5692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:14:26.454787 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 5692
I20240503 14:14:26.454998 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:14:26.458736 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.67:0
--local_ip_for_outbound_sockets=127.22.242.67
--webserver_interface=127.22.242.67
--webserver_port=0
--tserver_master_addrs=127.22.242.126:41785
--builtin_ntp_servers=127.22.242.84:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:14:26.466909 5819 heartbeater.cc:347] Connected to a master server at 127.22.242.126:41785
I20240503 14:14:26.467202 5819 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:26.467974 5819 heartbeater.cc:511] Master 127.22.242.126:41785 requested a full tablet report, sending...
I20240503 14:14:26.514945 5500 ts_manager.cc:194] Registered new tserver with Master: 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071)
I20240503 14:14:26.516239 5500 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.66:43515
W20240503 14:14:26.796475 5822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:26.796762 5822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:26.796849 5822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:26.804759 5822 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:26.804885 5822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:26.805008 5822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.67
I20240503 14:14:26.814294 5822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.242.67
--webserver_port=0
--tserver_master_addrs=127.22.242.126:41785
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.67
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:26.815557 5822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:26.817298 5822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:26.828805 5829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:26.828778 5828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:26.828878 5831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:26.829865 5822 server_base.cc:1034] running on GCE node
I20240503 14:14:26.830403 5822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:26.831286 5822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:26.832476 5822 hybrid_clock.cc:648] HybridClock initialized: now 1714745666832421 us; error 47 us; skew 500 ppm
I20240503 14:14:26.832780 5822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:26.834877 5822 webserver.cc:460] Webserver started at http://127.22.242.67:39675/ using document root <none> and password file <none>
I20240503 14:14:26.835574 5822 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:26.835704 5822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:26.836006 5822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:26.838548 5822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a8a749bf14364f22a4db0f7cdfa33ce9"
format_stamp: "Formatted at 2024-05-03 14:14:26 on dist-test-slave-r3pd"
I20240503 14:14:26.839291 5822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a8a749bf14364f22a4db0f7cdfa33ce9"
format_stamp: "Formatted at 2024-05-03 14:14:26 on dist-test-slave-r3pd"
I20240503 14:14:26.844623 5822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240503 14:14:26.847661 5837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:26.848924 5822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:14:26.849076 5822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a8a749bf14364f22a4db0f7cdfa33ce9"
format_stamp: "Formatted at 2024-05-03 14:14:26 on dist-test-slave-r3pd"
I20240503 14:14:26.849277 5822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:26.884389 5822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:26.885238 5822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:26.885521 5822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:26.886438 5822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:26.887936 5822 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:26.888060 5822 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:26.888193 5822 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:26.888268 5822 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240503 14:14:26.920946 5822 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.67:44189
I20240503 14:14:26.921005 5949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.67:44189 every 8 connection(s)
I20240503 14:14:26.922330 5822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:14:26.931941 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 5822
I20240503 14:14:26.932109 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:14:26.934649 5950 heartbeater.cc:347] Connected to a master server at 127.22.242.126:41785
I20240503 14:14:26.934937 5950 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:26.935287 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.68:0
--local_ip_for_outbound_sockets=127.22.242.68
--webserver_interface=127.22.242.68
--webserver_port=0
--tserver_master_addrs=127.22.242.126:41785
--builtin_ntp_servers=127.22.242.84:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:14:26.935760 5950 heartbeater.cc:511] Master 127.22.242.126:41785 requested a full tablet report, sending...
I20240503 14:14:26.998275 5500 ts_manager.cc:194] Registered new tserver with Master: a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
I20240503 14:14:26.999250 5500 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.67:36597
I20240503 14:14:27.049158 5687 heartbeater.cc:503] Master 127.22.242.126:41785 was elected leader, sending a full tablet report...
W20240503 14:14:27.288035 5954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:27.288379 5954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:27.288527 5954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:27.296289 5954 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:27.296430 5954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:27.296603 5954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.68
I20240503 14:14:27.305778 5954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.242.68
--webserver_port=0
--tserver_master_addrs=127.22.242.126:41785
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.68
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:27.307181 5954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:27.308974 5954 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:27.321082 5962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:27.321015 5959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:27.321133 5960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:27.321936 5954 server_base.cc:1034] running on GCE node
I20240503 14:14:27.322535 5954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:27.323477 5954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:27.324685 5954 hybrid_clock.cc:648] HybridClock initialized: now 1714745667324636 us; error 45 us; skew 500 ppm
I20240503 14:14:27.325002 5954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:27.330961 5954 webserver.cc:460] Webserver started at http://127.22.242.68:42357/ using document root <none> and password file <none>
I20240503 14:14:27.331732 5954 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:27.331863 5954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:27.332239 5954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:27.334884 5954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "da201109e13f4b36b3fba7477454a4c6"
format_stamp: "Formatted at 2024-05-03 14:14:27 on dist-test-slave-r3pd"
I20240503 14:14:27.335666 5954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "da201109e13f4b36b3fba7477454a4c6"
format_stamp: "Formatted at 2024-05-03 14:14:27 on dist-test-slave-r3pd"
I20240503 14:14:27.340883 5954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240503 14:14:27.344363 5968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:27.345713 5954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:14:27.345860 5954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "da201109e13f4b36b3fba7477454a4c6"
format_stamp: "Formatted at 2024-05-03 14:14:27 on dist-test-slave-r3pd"
I20240503 14:14:27.346043 5954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:27.363973 5954 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:27.364816 5954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:27.365219 5954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:27.366206 5954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:27.367679 5954 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:27.367786 5954 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:27.367897 5954 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:27.368006 5954 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:27.400552 5954 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.68:42505
I20240503 14:14:27.400594 6080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.68:42505 every 8 connection(s)
I20240503 14:14:27.401837 5954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:14:27.402242 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 5954
I20240503 14:14:27.402470 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:14:27.405984 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.69:0
--local_ip_for_outbound_sockets=127.22.242.69
--webserver_interface=127.22.242.69
--webserver_port=0
--tserver_master_addrs=127.22.242.126:41785
--builtin_ntp_servers=127.22.242.84:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:14:27.415980 6081 heartbeater.cc:347] Connected to a master server at 127.22.242.126:41785
I20240503 14:14:27.416337 6081 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:27.417200 6081 heartbeater.cc:511] Master 127.22.242.126:41785 requested a full tablet report, sending...
I20240503 14:14:27.468961 5500 ts_manager.cc:194] Registered new tserver with Master: da201109e13f4b36b3fba7477454a4c6 (127.22.242.68:42505)
I20240503 14:14:27.470494 5500 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.68:56005
I20240503 14:14:27.519105 5819 heartbeater.cc:503] Master 127.22.242.126:41785 was elected leader, sending a full tablet report...
W20240503 14:14:27.763959 6084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:27.764254 6084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:27.764346 6084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:27.771756 6084 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:27.771873 6084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:27.771993 6084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.69
I20240503 14:14:27.780768 6084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.242.69
--webserver_port=0
--tserver_master_addrs=127.22.242.126:41785
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.69
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:27.782011 6084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:27.783654 6084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:27.795194 6091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:27.795230 6093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:27.795190 6090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:27.795950 6084 server_base.cc:1034] running on GCE node
I20240503 14:14:27.796557 6084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:27.797464 6084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:27.798662 6084 hybrid_clock.cc:648] HybridClock initialized: now 1714745667798601 us; error 59 us; skew 500 ppm
I20240503 14:14:27.798959 6084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:27.801165 6084 webserver.cc:460] Webserver started at http://127.22.242.69:41551/ using document root <none> and password file <none>
I20240503 14:14:27.801885 6084 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:27.802012 6084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:27.802320 6084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:27.805028 6084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a24048f65ee54156b569d4bf1f738368"
format_stamp: "Formatted at 2024-05-03 14:14:27 on dist-test-slave-r3pd"
I20240503 14:14:27.805812 6084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a24048f65ee54156b569d4bf1f738368"
format_stamp: "Formatted at 2024-05-03 14:14:27 on dist-test-slave-r3pd"
I20240503 14:14:27.811002 6084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:14:27.814586 6099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:27.816637 6084 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20240503 14:14:27.816900 6084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a24048f65ee54156b569d4bf1f738368"
format_stamp: "Formatted at 2024-05-03 14:14:27 on dist-test-slave-r3pd"
I20240503 14:14:27.817240 6084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:27.860291 6084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:27.861186 6084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:27.861471 6084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:27.862435 6084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:27.863986 6084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:27.864092 6084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:27.864229 6084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:27.864301 6084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:27.896477 6084 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.69:42145
I20240503 14:14:27.896564 6211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.69:42145 every 8 connection(s)
I20240503 14:14:27.897754 6084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:14:27.899706 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 6084
I20240503 14:14:27.899886 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:14:27.903349 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.70:0
--local_ip_for_outbound_sockets=127.22.242.70
--webserver_interface=127.22.242.70
--webserver_port=0
--tserver_master_addrs=127.22.242.126:41785
--builtin_ntp_servers=127.22.242.84:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:14:27.910985 6212 heartbeater.cc:347] Connected to a master server at 127.22.242.126:41785
I20240503 14:14:27.911304 6212 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:27.912349 6212 heartbeater.cc:511] Master 127.22.242.126:41785 requested a full tablet report, sending...
I20240503 14:14:27.956131 5500 ts_manager.cc:194] Registered new tserver with Master: a24048f65ee54156b569d4bf1f738368 (127.22.242.69:42145)
I20240503 14:14:27.957538 5500 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.69:49833
I20240503 14:14:28.001958 5950 heartbeater.cc:503] Master 127.22.242.126:41785 was elected leader, sending a full tablet report...
W20240503 14:14:28.240574 6215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:28.240895 6215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:28.241050 6215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:28.248564 6215 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:28.248693 6215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:28.248849 6215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.70
I20240503 14:14:28.258222 6215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.242.70
--webserver_port=0
--tserver_master_addrs=127.22.242.126:41785
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.70
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:28.259562 6215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:28.261333 6215 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:28.272794 6224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:28.272755 6222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:28.272724 6221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:28.273741 6215 server_base.cc:1034] running on GCE node
I20240503 14:14:28.274325 6215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:28.275208 6215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:28.276417 6215 hybrid_clock.cc:648] HybridClock initialized: now 1714745668276371 us; error 32 us; skew 500 ppm
I20240503 14:14:28.276825 6215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:28.279732 6215 webserver.cc:460] Webserver started at http://127.22.242.70:40941/ using document root <none> and password file <none>
I20240503 14:14:28.280611 6215 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:28.280731 6215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:28.281108 6215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:28.283651 6215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "7bb84965b41b41fa9be23175c82e0a3a"
format_stamp: "Formatted at 2024-05-03 14:14:28 on dist-test-slave-r3pd"
I20240503 14:14:28.284497 6215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "7bb84965b41b41fa9be23175c82e0a3a"
format_stamp: "Formatted at 2024-05-03 14:14:28 on dist-test-slave-r3pd"
I20240503 14:14:28.290185 6215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 14:14:28.293468 6230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:28.294766 6215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240503 14:14:28.294921 6215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "7bb84965b41b41fa9be23175c82e0a3a"
format_stamp: "Formatted at 2024-05-03 14:14:28 on dist-test-slave-r3pd"
I20240503 14:14:28.295123 6215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:28.310027 6215 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:28.310897 6215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:28.311178 6215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:28.312141 6215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:28.313776 6215 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:28.313874 6215 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:28.313988 6215 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:28.314059 6215 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:28.345239 6215 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.70:43863
I20240503 14:14:28.345314 6342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.70:43863 every 8 connection(s)
I20240503 14:14:28.346496 6215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:14:28.355005 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 6215
I20240503 14:14:28.355166 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 14:14:28.358968 6343 heartbeater.cc:347] Connected to a master server at 127.22.242.126:41785
I20240503 14:14:28.359277 6343 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:28.360059 6343 heartbeater.cc:511] Master 127.22.242.126:41785 requested a full tablet report, sending...
I20240503 14:14:28.407215 5500 ts_manager.cc:194] Registered new tserver with Master: 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70:43863)
I20240503 14:14:28.408188 5500 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.70:50015
I20240503 14:14:28.409137 23497 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 14:14:28.425392 23497 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 14:14:28.425550 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 6084
I20240503 14:14:28.434129 6207 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:28.473269 6081 heartbeater.cc:503] Master 127.22.242.126:41785 was elected leader, sending a full tablet report...
I20240503 14:14:28.516122 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 6084
I20240503 14:14:28.532626 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 6215
I20240503 14:14:28.540755 6338 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:28.621245 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 6215
I20240503 14:14:28.637868 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 5954
I20240503 14:14:28.645877 6076 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:28.725162 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 5954
I20240503 14:14:32.493187 5501 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33358:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:14:32.499878 5501 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:14:32.533833 5884 tablet_service.cc:1467] Processing CreateTablet for tablet bfd4f9dbf8a946c6bb2c2b37210efe55 (DEFAULT_TABLE table=rebalance_test_table_0 [id=49c4626b38eb4b7194cd2add19c0cb32]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:32.533402 5752 tablet_service.cc:1467] Processing CreateTablet for tablet 7b8188059098481e9dd4b46858bda06c (DEFAULT_TABLE table=rebalance_test_table_0 [id=49c4626b38eb4b7194cd2add19c0cb32]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:32.534574 5885 tablet_service.cc:1467] Processing CreateTablet for tablet f6b542b55eb9477ab472b3a4005e6c63 (DEFAULT_TABLE table=rebalance_test_table_0 [id=49c4626b38eb4b7194cd2add19c0cb32]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:32.535022 5884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd4f9dbf8a946c6bb2c2b37210efe55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:32.535022 5752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b8188059098481e9dd4b46858bda06c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:32.534803 5622 tablet_service.cc:1467] Processing CreateTablet for tablet f6b542b55eb9477ab472b3a4005e6c63 (DEFAULT_TABLE table=rebalance_test_table_0 [id=49c4626b38eb4b7194cd2add19c0cb32]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:32.535679 5885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b542b55eb9477ab472b3a4005e6c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:32.536108 5622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b542b55eb9477ab472b3a4005e6c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:32.533669 5754 tablet_service.cc:1467] Processing CreateTablet for tablet f6b542b55eb9477ab472b3a4005e6c63 (DEFAULT_TABLE table=rebalance_test_table_0 [id=49c4626b38eb4b7194cd2add19c0cb32]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:32.540581 5754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b542b55eb9477ab472b3a4005e6c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:32.533397 5883 tablet_service.cc:1467] Processing CreateTablet for tablet 7b8188059098481e9dd4b46858bda06c (DEFAULT_TABLE table=rebalance_test_table_0 [id=49c4626b38eb4b7194cd2add19c0cb32]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:32.541142 5883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b8188059098481e9dd4b46858bda06c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:32.544667 5753 tablet_service.cc:1467] Processing CreateTablet for tablet bfd4f9dbf8a946c6bb2c2b37210efe55 (DEFAULT_TABLE table=rebalance_test_table_0 [id=49c4626b38eb4b7194cd2add19c0cb32]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:32.545243 5753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd4f9dbf8a946c6bb2c2b37210efe55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:32.557902 5620 tablet_service.cc:1467] Processing CreateTablet for tablet 7b8188059098481e9dd4b46858bda06c (DEFAULT_TABLE table=rebalance_test_table_0 [id=49c4626b38eb4b7194cd2add19c0cb32]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:32.558811 5620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b8188059098481e9dd4b46858bda06c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:32.559897 5621 tablet_service.cc:1467] Processing CreateTablet for tablet bfd4f9dbf8a946c6bb2c2b37210efe55 (DEFAULT_TABLE table=rebalance_test_table_0 [id=49c4626b38eb4b7194cd2add19c0cb32]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:32.560022 6370 tablet_bootstrap.cc:492] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9: Bootstrap starting.
I20240503 14:14:32.560559 5621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd4f9dbf8a946c6bb2c2b37210efe55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:32.563810 6370 tablet_bootstrap.cc:654] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:32.565225 6370 log.cc:824] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:32.579748 6369 tablet_bootstrap.cc:492] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f: Bootstrap starting.
I20240503 14:14:32.583137 6370 tablet_bootstrap.cc:492] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9: No bootstrap required, opened a new log
I20240503 14:14:32.583451 6370 ts_tablet_manager.cc:1367] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.014s
I20240503 14:14:32.583652 6369 tablet_bootstrap.cc:654] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:32.584386 6372 tablet_bootstrap.cc:492] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33: Bootstrap starting.
I20240503 14:14:32.585013 6369 log.cc:824] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:32.587316 6372 tablet_bootstrap.cc:654] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:32.588007 6369 tablet_bootstrap.cc:492] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f: No bootstrap required, opened a new log
I20240503 14:14:32.588307 6369 ts_tablet_manager.cc:1367] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20240503 14:14:32.588410 6372 log.cc:824] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:32.591003 6372 tablet_bootstrap.cc:492] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33: No bootstrap required, opened a new log
I20240503 14:14:32.591233 6372 ts_tablet_manager.cc:1367] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240503 14:14:32.592571 6370 raft_consensus.cc:348] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.593030 6370 raft_consensus.cc:374] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:32.593191 6370 raft_consensus.cc:729] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a749bf14364f22a4db0f7cdfa33ce9, State: Initialized, Role: FOLLOWER
I20240503 14:14:32.593910 6370 consensus_queue.cc:260] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.595646 6370 ts_tablet_manager.cc:1398] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent starting tablet: real 0.012s user 0.004s sys 0.007s
I20240503 14:14:32.596065 6370 tablet_bootstrap.cc:492] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9: Bootstrap starting.
I20240503 14:14:32.597816 6370 tablet_bootstrap.cc:654] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:32.598596 6372 raft_consensus.cc:348] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.598587 6369 raft_consensus.cc:348] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.599112 6369 raft_consensus.cc:374] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:32.599170 6372 raft_consensus.cc:374] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:32.599306 6369 raft_consensus.cc:729] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5361a07e89ad47298aabe717a4f2867f, State: Initialized, Role: FOLLOWER
I20240503 14:14:32.599334 6372 raft_consensus.cc:729] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5237ac15b5c4a6c88a8a07f3d44cf33, State: Initialized, Role: FOLLOWER
I20240503 14:14:32.600103 6372 consensus_queue.cc:260] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.600347 6369 consensus_queue.cc:260] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.606261 6372 ts_tablet_manager.cc:1398] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20240503 14:14:32.606356 6370 tablet_bootstrap.cc:492] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9: No bootstrap required, opened a new log
I20240503 14:14:32.606556 6370 ts_tablet_manager.cc:1367] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240503 14:14:32.606794 6372 tablet_bootstrap.cc:492] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: Bootstrap starting.
I20240503 14:14:32.608641 6372 tablet_bootstrap.cc:654] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:32.608633 6369 ts_tablet_manager.cc:1398] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f: Time spent starting tablet: real 0.020s user 0.012s sys 0.005s
I20240503 14:14:32.609045 6369 tablet_bootstrap.cc:492] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f: Bootstrap starting.
I20240503 14:14:32.610934 6369 tablet_bootstrap.cc:654] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:32.611137 6370 raft_consensus.cc:348] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.611435 6370 raft_consensus.cc:374] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:32.611574 6370 raft_consensus.cc:729] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a749bf14364f22a4db0f7cdfa33ce9, State: Initialized, Role: FOLLOWER
I20240503 14:14:32.611990 6370 consensus_queue.cc:260] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.612771 6370 ts_tablet_manager.cc:1398] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:14:32.613277 6370 tablet_bootstrap.cc:492] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9: Bootstrap starting.
I20240503 14:14:32.615525 6370 tablet_bootstrap.cc:654] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:32.616938 6372 tablet_bootstrap.cc:492] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: No bootstrap required, opened a new log
I20240503 14:14:32.617146 6372 ts_tablet_manager.cc:1367] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20240503 14:14:32.618264 6372 raft_consensus.cc:348] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.618538 6372 raft_consensus.cc:374] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:32.618683 6372 raft_consensus.cc:729] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5237ac15b5c4a6c88a8a07f3d44cf33, State: Initialized, Role: FOLLOWER
I20240503 14:14:32.618981 6372 consensus_queue.cc:260] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.619282 6369 tablet_bootstrap.cc:492] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f: No bootstrap required, opened a new log
I20240503 14:14:32.619452 6369 ts_tablet_manager.cc:1367] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20240503 14:14:32.619546 6372 ts_tablet_manager.cc:1398] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:32.619861 6372 tablet_bootstrap.cc:492] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33: Bootstrap starting.
I20240503 14:14:32.620770 6369 raft_consensus.cc:348] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.621124 6369 raft_consensus.cc:374] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:32.621279 6369 raft_consensus.cc:729] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5361a07e89ad47298aabe717a4f2867f, State: Initialized, Role: FOLLOWER
I20240503 14:14:32.621538 6372 tablet_bootstrap.cc:654] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:32.621695 6369 consensus_queue.cc:260] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.622550 6369 ts_tablet_manager.cc:1398] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:14:32.622942 6369 tablet_bootstrap.cc:492] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f: Bootstrap starting.
I20240503 14:14:32.624207 6370 tablet_bootstrap.cc:492] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9: No bootstrap required, opened a new log
I20240503 14:14:32.624420 6370 ts_tablet_manager.cc:1367] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240503 14:14:32.624635 6369 tablet_bootstrap.cc:654] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:32.625223 6370 raft_consensus.cc:348] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.625505 6370 raft_consensus.cc:374] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:32.625665 6370 raft_consensus.cc:729] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a749bf14364f22a4db0f7cdfa33ce9, State: Initialized, Role: FOLLOWER
I20240503 14:14:32.625967 6370 consensus_queue.cc:260] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.626761 6370 ts_tablet_manager.cc:1398] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:32.628150 6369 tablet_bootstrap.cc:492] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f: No bootstrap required, opened a new log
I20240503 14:14:32.628398 6369 ts_tablet_manager.cc:1367] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240503 14:14:32.628965 6372 tablet_bootstrap.cc:492] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33: No bootstrap required, opened a new log
I20240503 14:14:32.628998 6369 raft_consensus.cc:348] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.629135 6372 ts_tablet_manager.cc:1367] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 14:14:32.629215 6369 raft_consensus.cc:374] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:32.629298 6369 raft_consensus.cc:729] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5361a07e89ad47298aabe717a4f2867f, State: Initialized, Role: FOLLOWER
I20240503 14:14:32.629525 6369 consensus_queue.cc:260] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.630074 6369 ts_tablet_manager.cc:1398] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:14:32.630259 6372 raft_consensus.cc:348] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.630435 6372 raft_consensus.cc:374] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:32.630509 6372 raft_consensus.cc:729] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5237ac15b5c4a6c88a8a07f3d44cf33, State: Initialized, Role: FOLLOWER
I20240503 14:14:32.630787 6372 consensus_queue.cc:260] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.631415 6372 ts_tablet_manager.cc:1398] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240503 14:14:32.678369 5951 tablet.cc:2369] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:14:32.698398 6377 raft_consensus.cc:482] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:32.698627 6377 raft_consensus.cc:504] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.700352 6377 leader_election.cc:288] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071), a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
W20240503 14:14:32.708685 5820 tablet.cc:2369] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:14:32.709221 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8188059098481e9dd4b46858bda06c" candidate_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" is_pre_election: true
I20240503 14:14:32.709808 5905 raft_consensus.cc:2454] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5237ac15b5c4a6c88a8a07f3d44cf33 in term 0.
I20240503 14:14:32.710390 5575 leader_election.cc:302] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8a749bf14364f22a4db0f7cdfa33ce9, e5237ac15b5c4a6c88a8a07f3d44cf33; no voters:
I20240503 14:14:32.710431 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8188059098481e9dd4b46858bda06c" candidate_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5361a07e89ad47298aabe717a4f2867f" is_pre_election: true
I20240503 14:14:32.710911 6377 raft_consensus.cc:2781] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:32.710953 5774 raft_consensus.cc:2454] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5237ac15b5c4a6c88a8a07f3d44cf33 in term 0.
I20240503 14:14:32.711081 6377 raft_consensus.cc:482] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:32.711203 6377 raft_consensus.cc:3037] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:32.713519 6377 raft_consensus.cc:504] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.714140 6377 leader_election.cc:288] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [CANDIDATE]: Term 1 election: Requested vote from peers 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071), a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
I20240503 14:14:32.714509 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8188059098481e9dd4b46858bda06c" candidate_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5361a07e89ad47298aabe717a4f2867f"
I20240503 14:14:32.714637 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8188059098481e9dd4b46858bda06c" candidate_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9"
I20240503 14:14:32.714716 5774 raft_consensus.cc:3037] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:32.714841 5905 raft_consensus.cc:3037] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:14:32.717715 5688 tablet.cc:2369] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:14:32.717800 5905 raft_consensus.cc:2454] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5237ac15b5c4a6c88a8a07f3d44cf33 in term 1.
I20240503 14:14:32.717800 5774 raft_consensus.cc:2454] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5237ac15b5c4a6c88a8a07f3d44cf33 in term 1.
I20240503 14:14:32.718190 5577 leader_election.cc:302] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5361a07e89ad47298aabe717a4f2867f, e5237ac15b5c4a6c88a8a07f3d44cf33; no voters:
I20240503 14:14:32.718516 6377 raft_consensus.cc:2781] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:32.718926 6377 raft_consensus.cc:686] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Becoming Leader. State: Replica: e5237ac15b5c4a6c88a8a07f3d44cf33, State: Running, Role: LEADER
I20240503 14:14:32.719364 6377 consensus_queue.cc:237] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.723726 5500 catalog_manager.cc:5484] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:32.749897 6375 raft_consensus.cc:482] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:32.750118 6375 raft_consensus.cc:504] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.751786 6375 leader_election.cc:288] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157), 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071)
I20240503 14:14:32.758805 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b542b55eb9477ab472b3a4005e6c63" candidate_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5361a07e89ad47298aabe717a4f2867f" is_pre_election: true
I20240503 14:14:32.758913 5642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b542b55eb9477ab472b3a4005e6c63" candidate_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" is_pre_election: true
I20240503 14:14:32.759114 5774 raft_consensus.cc:2454] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8a749bf14364f22a4db0f7cdfa33ce9 in term 0.
I20240503 14:14:32.759348 5642 raft_consensus.cc:2454] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8a749bf14364f22a4db0f7cdfa33ce9 in term 0.
I20240503 14:14:32.759589 5840 leader_election.cc:302] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5361a07e89ad47298aabe717a4f2867f, a8a749bf14364f22a4db0f7cdfa33ce9; no voters:
I20240503 14:14:32.759717 6377 raft_consensus.cc:482] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:32.759874 6377 raft_consensus.cc:504] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.760016 6375 raft_consensus.cc:2781] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:32.760226 6375 raft_consensus.cc:482] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:32.760325 6375 raft_consensus.cc:3037] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:32.760548 6377 leader_election.cc:288] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071), a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
I20240503 14:14:32.761015 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd4f9dbf8a946c6bb2c2b37210efe55" candidate_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5361a07e89ad47298aabe717a4f2867f" is_pre_election: true
I20240503 14:14:32.761231 5774 raft_consensus.cc:2454] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5237ac15b5c4a6c88a8a07f3d44cf33 in term 0.
I20240503 14:14:32.761286 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd4f9dbf8a946c6bb2c2b37210efe55" candidate_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" is_pre_election: true
I20240503 14:14:32.761523 5905 raft_consensus.cc:2454] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5237ac15b5c4a6c88a8a07f3d44cf33 in term 0.
I20240503 14:14:32.761597 5577 leader_election.cc:302] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5361a07e89ad47298aabe717a4f2867f, e5237ac15b5c4a6c88a8a07f3d44cf33; no voters:
I20240503 14:14:32.761860 6377 raft_consensus.cc:2781] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:32.761988 6377 raft_consensus.cc:482] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:32.762148 6377 raft_consensus.cc:3037] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:32.763113 6375 raft_consensus.cc:504] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.763809 6375 leader_election.cc:288] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [CANDIDATE]: Term 1 election: Requested vote from peers e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157), 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071)
I20240503 14:14:32.764194 5642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b542b55eb9477ab472b3a4005e6c63" candidate_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33"
I20240503 14:14:32.764237 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b542b55eb9477ab472b3a4005e6c63" candidate_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5361a07e89ad47298aabe717a4f2867f"
I20240503 14:14:32.764396 5642 raft_consensus.cc:3037] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:32.764446 5774 raft_consensus.cc:3037] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:32.764648 6377 raft_consensus.cc:504] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.765342 6377 leader_election.cc:288] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [CANDIDATE]: Term 1 election: Requested vote from peers 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071), a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
I20240503 14:14:32.765914 5773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd4f9dbf8a946c6bb2c2b37210efe55" candidate_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5361a07e89ad47298aabe717a4f2867f"
I20240503 14:14:32.766016 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd4f9dbf8a946c6bb2c2b37210efe55" candidate_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9"
I20240503 14:14:32.766176 5773 raft_consensus.cc:3037] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:32.766209 5905 raft_consensus.cc:3037] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:32.767097 5774 raft_consensus.cc:2454] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8a749bf14364f22a4db0f7cdfa33ce9 in term 1.
I20240503 14:14:32.767290 5642 raft_consensus.cc:2454] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8a749bf14364f22a4db0f7cdfa33ce9 in term 1.
I20240503 14:14:32.767488 5840 leader_election.cc:302] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5361a07e89ad47298aabe717a4f2867f, a8a749bf14364f22a4db0f7cdfa33ce9; no voters:
I20240503 14:14:32.767767 6375 raft_consensus.cc:2781] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:32.768616 6375 raft_consensus.cc:686] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Becoming Leader. State: Replica: a8a749bf14364f22a4db0f7cdfa33ce9, State: Running, Role: LEADER
I20240503 14:14:32.768709 5773 raft_consensus.cc:2454] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5237ac15b5c4a6c88a8a07f3d44cf33 in term 1.
I20240503 14:14:32.769191 6375 consensus_queue.cc:237] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.769398 5577 leader_election.cc:302] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5361a07e89ad47298aabe717a4f2867f, e5237ac15b5c4a6c88a8a07f3d44cf33; no voters:
I20240503 14:14:32.769654 6377 raft_consensus.cc:2781] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:32.769666 5905 raft_consensus.cc:2454] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5237ac15b5c4a6c88a8a07f3d44cf33 in term 1.
I20240503 14:14:32.769889 6377 raft_consensus.cc:686] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Becoming Leader. State: Replica: e5237ac15b5c4a6c88a8a07f3d44cf33, State: Running, Role: LEADER
I20240503 14:14:32.770138 6377 consensus_queue.cc:237] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:32.772727 5500 catalog_manager.cc:5484] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:32.774893 5499 catalog_manager.cc:5484] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67). New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } health_report { overall_health: HEALTHY } } }
W20240503 14:14:33.153025 6393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:33.153383 6393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:33.160956 6393 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:14:33.199595 6377 consensus_queue.cc:1035] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240503 14:14:33.208025 6386 consensus_queue.cc:1035] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:33.297952 6375 consensus_queue.cc:1035] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:33.310448 5905 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:33.311089 6407 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:33.312290 5773 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:33.312980 6377 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:33.333585 6432 mvcc.cc:205] Tried to move back new op lower bound from 7023598277868994560 to 7023598275668512768. Current Snapshot: MvccSnapshot[applied={T|T < 7023598277868994560}]
I20240503 14:14:33.337090 6375 consensus_queue.cc:1035] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:33.406210 6406 mvcc.cc:205] Tried to move back new op lower bound from 7023598277883105280 to 7023598275667939328. Current Snapshot: MvccSnapshot[applied={T|T < 7023598277883105280}]
I20240503 14:14:33.453344 6448 mvcc.cc:205] Tried to move back new op lower bound from 7023598277868994560 to 7023598275668512768. Current Snapshot: MvccSnapshot[applied={T|T < 7023598277868994560}]
INSERT report
rows total: 2000
time total: 261.492 ms
time per row: 0.130746 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:14:34.555460 5498 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33358:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:14:34.556306 5498 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:14:34.564630 5752 tablet_service.cc:1467] Processing CreateTablet for tablet 614d4da74aa547a28013c97f5a1079fd (DEFAULT_TABLE table=rebalance_test_table_1 [id=a268a544233045a1b8bcb56727a9bbb8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:34.565172 5752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 614d4da74aa547a28013c97f5a1079fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:34.565133 5884 tablet_service.cc:1467] Processing CreateTablet for tablet 614d4da74aa547a28013c97f5a1079fd (DEFAULT_TABLE table=rebalance_test_table_1 [id=a268a544233045a1b8bcb56727a9bbb8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:34.565528 5884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 614d4da74aa547a28013c97f5a1079fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:34.566267 5622 tablet_service.cc:1467] Processing CreateTablet for tablet 52983271755542f090bb26d0675ec84e (DEFAULT_TABLE table=rebalance_test_table_1 [id=a268a544233045a1b8bcb56727a9bbb8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:34.566653 5622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52983271755542f090bb26d0675ec84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:34.568634 5883 tablet_service.cc:1467] Processing CreateTablet for tablet eb339ed8f9a141caa62cc5588d07fee1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a268a544233045a1b8bcb56727a9bbb8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:34.569052 5883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb339ed8f9a141caa62cc5588d07fee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:34.572018 5754 tablet_service.cc:1467] Processing CreateTablet for tablet 52983271755542f090bb26d0675ec84e (DEFAULT_TABLE table=rebalance_test_table_1 [id=a268a544233045a1b8bcb56727a9bbb8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:34.572448 5754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52983271755542f090bb26d0675ec84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:34.572654 5885 tablet_service.cc:1467] Processing CreateTablet for tablet 52983271755542f090bb26d0675ec84e (DEFAULT_TABLE table=rebalance_test_table_1 [id=a268a544233045a1b8bcb56727a9bbb8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:34.573137 5885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52983271755542f090bb26d0675ec84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:34.575183 6460 tablet_bootstrap.cc:492] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: Bootstrap starting.
I20240503 14:14:34.576643 5753 tablet_service.cc:1467] Processing CreateTablet for tablet eb339ed8f9a141caa62cc5588d07fee1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a268a544233045a1b8bcb56727a9bbb8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:34.576817 5620 tablet_service.cc:1467] Processing CreateTablet for tablet 614d4da74aa547a28013c97f5a1079fd (DEFAULT_TABLE table=rebalance_test_table_1 [id=a268a544233045a1b8bcb56727a9bbb8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:34.576939 6460 tablet_bootstrap.cc:654] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:34.577158 5753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb339ed8f9a141caa62cc5588d07fee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:34.577247 5620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 614d4da74aa547a28013c97f5a1079fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:34.580703 5621 tablet_service.cc:1467] Processing CreateTablet for tablet eb339ed8f9a141caa62cc5588d07fee1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a268a544233045a1b8bcb56727a9bbb8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:34.581213 5621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb339ed8f9a141caa62cc5588d07fee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:34.581957 6459 tablet_bootstrap.cc:492] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f: Bootstrap starting.
I20240503 14:14:34.584106 6459 tablet_bootstrap.cc:654] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:34.593305 6459 tablet_bootstrap.cc:492] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f: No bootstrap required, opened a new log
I20240503 14:14:34.594121 6459 ts_tablet_manager.cc:1367] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240503 14:14:34.595193 6459 raft_consensus.cc:348] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.596453 6459 raft_consensus.cc:374] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:34.596731 6459 raft_consensus.cc:729] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5361a07e89ad47298aabe717a4f2867f, State: Initialized, Role: FOLLOWER
I20240503 14:14:34.597179 6459 consensus_queue.cc:260] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.598914 6461 tablet_bootstrap.cc:492] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33: Bootstrap starting.
I20240503 14:14:34.599757 6459 ts_tablet_manager.cc:1398] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:14:34.600098 6459 tablet_bootstrap.cc:492] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f: Bootstrap starting.
I20240503 14:14:34.600695 6461 tablet_bootstrap.cc:654] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:34.601737 6459 tablet_bootstrap.cc:654] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:34.610332 6459 tablet_bootstrap.cc:492] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f: No bootstrap required, opened a new log
I20240503 14:14:34.610683 6459 ts_tablet_manager.cc:1367] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20240503 14:14:34.611811 6459 raft_consensus.cc:348] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.612252 6459 raft_consensus.cc:374] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:34.612406 6459 raft_consensus.cc:729] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5361a07e89ad47298aabe717a4f2867f, State: Initialized, Role: FOLLOWER
I20240503 14:14:34.612789 6459 consensus_queue.cc:260] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.613957 6459 ts_tablet_manager.cc:1398] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:14:34.614511 6459 tablet_bootstrap.cc:492] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f: Bootstrap starting.
I20240503 14:14:34.616704 6459 tablet_bootstrap.cc:654] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:34.623924 6460 tablet_bootstrap.cc:492] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: No bootstrap required, opened a new log
I20240503 14:14:34.624145 6460 ts_tablet_manager.cc:1367] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent bootstrapping tablet: real 0.049s user 0.012s sys 0.032s
I20240503 14:14:34.624404 6461 tablet_bootstrap.cc:492] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33: No bootstrap required, opened a new log
I20240503 14:14:34.624550 6461 ts_tablet_manager.cc:1367] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.016s
I20240503 14:14:34.624621 6459 tablet_bootstrap.cc:492] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f: No bootstrap required, opened a new log
I20240503 14:14:34.624775 6459 ts_tablet_manager.cc:1367] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.001s
I20240503 14:14:34.625000 6460 raft_consensus.cc:348] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.625218 6460 raft_consensus.cc:374] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:34.625212 6461 raft_consensus.cc:348] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.625377 6460 raft_consensus.cc:729] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a749bf14364f22a4db0f7cdfa33ce9, State: Initialized, Role: FOLLOWER
I20240503 14:14:34.625420 6461 raft_consensus.cc:374] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:34.625516 6461 raft_consensus.cc:729] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5237ac15b5c4a6c88a8a07f3d44cf33, State: Initialized, Role: FOLLOWER
I20240503 14:14:34.625546 6459 raft_consensus.cc:348] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.625768 6459 raft_consensus.cc:374] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:34.625644 6460 consensus_queue.cc:260] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.625860 6459 raft_consensus.cc:729] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5361a07e89ad47298aabe717a4f2867f, State: Initialized, Role: FOLLOWER
I20240503 14:14:34.625783 6461 consensus_queue.cc:260] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.626246 6461 ts_tablet_manager.cc:1398] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:14:34.626169 6459 consensus_queue.cc:260] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.626644 6461 tablet_bootstrap.cc:492] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33: Bootstrap starting.
I20240503 14:14:34.626742 6459 ts_tablet_manager.cc:1398] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:14:34.627504 6460 ts_tablet_manager.cc:1398] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:14:34.627810 6460 tablet_bootstrap.cc:492] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9: Bootstrap starting.
I20240503 14:14:34.628363 6461 tablet_bootstrap.cc:654] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:34.629515 6460 tablet_bootstrap.cc:654] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:34.631071 6460 tablet_bootstrap.cc:492] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9: No bootstrap required, opened a new log
I20240503 14:14:34.631446 6460 ts_tablet_manager.cc:1367] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:14:34.632247 6460 raft_consensus.cc:348] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.632668 6460 raft_consensus.cc:374] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:34.632802 6460 raft_consensus.cc:729] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a749bf14364f22a4db0f7cdfa33ce9, State: Initialized, Role: FOLLOWER
I20240503 14:14:34.633059 6460 consensus_queue.cc:260] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.633605 6460 ts_tablet_manager.cc:1398] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:34.633854 6460 tablet_bootstrap.cc:492] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: Bootstrap starting.
I20240503 14:14:34.635412 6461 tablet_bootstrap.cc:492] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33: No bootstrap required, opened a new log
I20240503 14:14:34.635447 6460 tablet_bootstrap.cc:654] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:34.635593 6461 ts_tablet_manager.cc:1367] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20240503 14:14:34.636448 6461 raft_consensus.cc:348] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.636677 6461 raft_consensus.cc:374] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:34.636744 6461 raft_consensus.cc:729] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5237ac15b5c4a6c88a8a07f3d44cf33, State: Initialized, Role: FOLLOWER
I20240503 14:14:34.636940 6461 consensus_queue.cc:260] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.637056 6460 tablet_bootstrap.cc:492] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: No bootstrap required, opened a new log
I20240503 14:14:34.637212 6460 ts_tablet_manager.cc:1367] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20240503 14:14:34.637419 6461 ts_tablet_manager.cc:1398] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:14:34.637717 6461 tablet_bootstrap.cc:492] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33: Bootstrap starting.
I20240503 14:14:34.637910 6460 raft_consensus.cc:348] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.638113 6460 raft_consensus.cc:374] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:34.638238 6460 raft_consensus.cc:729] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a749bf14364f22a4db0f7cdfa33ce9, State: Initialized, Role: FOLLOWER
I20240503 14:14:34.638465 6460 consensus_queue.cc:260] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.639019 6460 ts_tablet_manager.cc:1398] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:14:34.639266 6461 tablet_bootstrap.cc:654] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:34.641280 6461 tablet_bootstrap.cc:492] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33: No bootstrap required, opened a new log
I20240503 14:14:34.641431 6461 ts_tablet_manager.cc:1367] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:14:34.642288 6461 raft_consensus.cc:348] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.642578 6461 raft_consensus.cc:374] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:34.642737 6461 raft_consensus.cc:729] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5237ac15b5c4a6c88a8a07f3d44cf33, State: Initialized, Role: FOLLOWER
I20240503 14:14:34.643025 6461 consensus_queue.cc:260] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.643528 6461 ts_tablet_manager.cc:1398] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:14:34.646355 6464 raft_consensus.cc:482] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:34.646566 6464 raft_consensus.cc:504] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.648229 6464 leader_election.cc:288] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157), a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
I20240503 14:14:34.654352 5642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "614d4da74aa547a28013c97f5a1079fd" candidate_uuid: "5361a07e89ad47298aabe717a4f2867f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" is_pre_election: true
I20240503 14:14:34.654623 5642 raft_consensus.cc:2454] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5361a07e89ad47298aabe717a4f2867f in term 0.
I20240503 14:14:34.655148 5707 leader_election.cc:302] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5361a07e89ad47298aabe717a4f2867f, e5237ac15b5c4a6c88a8a07f3d44cf33; no voters:
I20240503 14:14:34.655364 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "614d4da74aa547a28013c97f5a1079fd" candidate_uuid: "5361a07e89ad47298aabe717a4f2867f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" is_pre_election: true
I20240503 14:14:34.655580 6464 raft_consensus.cc:2781] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:34.655601 5905 raft_consensus.cc:2454] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5361a07e89ad47298aabe717a4f2867f in term 0.
I20240503 14:14:34.655717 6464 raft_consensus.cc:482] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:34.655812 6464 raft_consensus.cc:3037] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:34.658010 6464 raft_consensus.cc:504] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.658565 6464 leader_election.cc:288] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [CANDIDATE]: Term 1 election: Requested vote from peers e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157), a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
I20240503 14:14:34.659021 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "614d4da74aa547a28013c97f5a1079fd" candidate_uuid: "5361a07e89ad47298aabe717a4f2867f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9"
I20240503 14:14:34.659021 5642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "614d4da74aa547a28013c97f5a1079fd" candidate_uuid: "5361a07e89ad47298aabe717a4f2867f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33"
I20240503 14:14:34.659255 5905 raft_consensus.cc:3037] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:34.659255 5642 raft_consensus.cc:3037] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:34.662067 5905 raft_consensus.cc:2454] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5361a07e89ad47298aabe717a4f2867f in term 1.
I20240503 14:14:34.662076 5642 raft_consensus.cc:2454] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5361a07e89ad47298aabe717a4f2867f in term 1.
I20240503 14:14:34.662451 5707 leader_election.cc:302] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5361a07e89ad47298aabe717a4f2867f, e5237ac15b5c4a6c88a8a07f3d44cf33; no voters:
I20240503 14:14:34.662729 6464 raft_consensus.cc:2781] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:34.663138 6464 raft_consensus.cc:686] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Becoming Leader. State: Replica: 5361a07e89ad47298aabe717a4f2867f, State: Running, Role: LEADER
I20240503 14:14:34.663619 6464 consensus_queue.cc:237] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.666831 5498 catalog_manager.cc:5484] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5361a07e89ad47298aabe717a4f2867f (127.22.242.66). New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:34.871543 6386 raft_consensus.cc:482] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:34.871800 6386 raft_consensus.cc:504] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.872506 6386 leader_election.cc:288] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071), a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
I20240503 14:14:34.873080 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb339ed8f9a141caa62cc5588d07fee1" candidate_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" is_pre_election: true
I20240503 14:14:34.873111 5772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb339ed8f9a141caa62cc5588d07fee1" candidate_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5361a07e89ad47298aabe717a4f2867f" is_pre_election: true
I20240503 14:14:34.873288 5905 raft_consensus.cc:2454] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5237ac15b5c4a6c88a8a07f3d44cf33 in term 0.
I20240503 14:14:34.873327 5772 raft_consensus.cc:2454] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5237ac15b5c4a6c88a8a07f3d44cf33 in term 0.
I20240503 14:14:34.873703 5575 leader_election.cc:302] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8a749bf14364f22a4db0f7cdfa33ce9, e5237ac15b5c4a6c88a8a07f3d44cf33; no voters:
I20240503 14:14:34.873996 6386 raft_consensus.cc:2781] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:34.874079 6386 raft_consensus.cc:482] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:34.874176 6386 raft_consensus.cc:3037] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:34.876065 6386 raft_consensus.cc:504] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.876725 6386 leader_election.cc:288] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [CANDIDATE]: Term 1 election: Requested vote from peers 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071), a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
I20240503 14:14:34.877148 5772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb339ed8f9a141caa62cc5588d07fee1" candidate_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5361a07e89ad47298aabe717a4f2867f"
I20240503 14:14:34.877247 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb339ed8f9a141caa62cc5588d07fee1" candidate_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9"
I20240503 14:14:34.877369 5772 raft_consensus.cc:3037] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:34.877427 5905 raft_consensus.cc:3037] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:34.880255 5772 raft_consensus.cc:2454] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5237ac15b5c4a6c88a8a07f3d44cf33 in term 1.
I20240503 14:14:34.880249 5905 raft_consensus.cc:2454] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5237ac15b5c4a6c88a8a07f3d44cf33 in term 1.
I20240503 14:14:34.880745 5577 leader_election.cc:302] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5361a07e89ad47298aabe717a4f2867f, a8a749bf14364f22a4db0f7cdfa33ce9, e5237ac15b5c4a6c88a8a07f3d44cf33; no voters:
I20240503 14:14:34.881040 6386 raft_consensus.cc:2781] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:34.881418 6386 raft_consensus.cc:686] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Becoming Leader. State: Replica: e5237ac15b5c4a6c88a8a07f3d44cf33, State: Running, Role: LEADER
I20240503 14:14:34.881685 6386 consensus_queue.cc:237] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.884332 5498 catalog_manager.cc:5484] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:34.998857 6464 raft_consensus.cc:482] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:34.999107 6464 raft_consensus.cc:504] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:34.999734 6464 leader_election.cc:288] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157), a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
I20240503 14:14:35.000238 5642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52983271755542f090bb26d0675ec84e" candidate_uuid: "5361a07e89ad47298aabe717a4f2867f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" is_pre_election: true
I20240503 14:14:35.000306 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52983271755542f090bb26d0675ec84e" candidate_uuid: "5361a07e89ad47298aabe717a4f2867f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" is_pre_election: true
I20240503 14:14:35.000450 5642 raft_consensus.cc:2454] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5361a07e89ad47298aabe717a4f2867f in term 0.
I20240503 14:14:35.000545 5905 raft_consensus.cc:2454] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5361a07e89ad47298aabe717a4f2867f in term 0.
I20240503 14:14:35.000866 5707 leader_election.cc:302] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5361a07e89ad47298aabe717a4f2867f, e5237ac15b5c4a6c88a8a07f3d44cf33; no voters:
I20240503 14:14:35.001152 6464 raft_consensus.cc:2781] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:35.001231 6464 raft_consensus.cc:482] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:35.001327 6464 raft_consensus.cc:3037] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:35.003213 6464 raft_consensus.cc:504] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:35.003818 6464 leader_election.cc:288] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [CANDIDATE]: Term 1 election: Requested vote from peers e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157), a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
I20240503 14:14:35.004325 5642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52983271755542f090bb26d0675ec84e" candidate_uuid: "5361a07e89ad47298aabe717a4f2867f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33"
I20240503 14:14:35.004355 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52983271755542f090bb26d0675ec84e" candidate_uuid: "5361a07e89ad47298aabe717a4f2867f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9"
I20240503 14:14:35.004518 5642 raft_consensus.cc:3037] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:35.004549 5905 raft_consensus.cc:3037] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:35.006414 5905 raft_consensus.cc:2454] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5361a07e89ad47298aabe717a4f2867f in term 1.
I20240503 14:14:35.006426 5642 raft_consensus.cc:2454] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5361a07e89ad47298aabe717a4f2867f in term 1.
I20240503 14:14:35.006867 5707 leader_election.cc:302] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5361a07e89ad47298aabe717a4f2867f, a8a749bf14364f22a4db0f7cdfa33ce9; no voters:
I20240503 14:14:35.007160 6464 raft_consensus.cc:2781] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:35.007320 6464 raft_consensus.cc:686] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Becoming Leader. State: Replica: 5361a07e89ad47298aabe717a4f2867f, State: Running, Role: LEADER
I20240503 14:14:35.007570 6464 consensus_queue.cc:237] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:35.010313 5498 catalog_manager.cc:5484] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5361a07e89ad47298aabe717a4f2867f (127.22.242.66). New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:35.193933 6464 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:35.199575 6464 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:35.296532 6478 consensus_queue.cc:1035] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:35.300654 6489 consensus_queue.cc:1035] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:14:35.390851 6481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:35.391211 6481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:35.398730 6481 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 14:14:35.519224 6476 consensus_queue.cc:1035] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:35.522528 5642 raft_consensus.cc:1261] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:35.523386 6476 consensus_queue.cc:1035] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 195.548 ms
time per row: 0.097774 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:14:36.660584 5496 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33358:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:14:36.661269 5496 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:14:36.669040 5620 tablet_service.cc:1467] Processing CreateTablet for tablet 098804415c3b4bafa53e58119712758f (DEFAULT_TABLE table=rebalance_test_table_2 [id=632b93f2c4174b0080e62682cf1200b4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:36.669260 5753 tablet_service.cc:1467] Processing CreateTablet for tablet 098804415c3b4bafa53e58119712758f (DEFAULT_TABLE table=rebalance_test_table_2 [id=632b93f2c4174b0080e62682cf1200b4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:36.669533 5620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 098804415c3b4bafa53e58119712758f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:36.669657 5753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 098804415c3b4bafa53e58119712758f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:36.669674 5883 tablet_service.cc:1467] Processing CreateTablet for tablet 098804415c3b4bafa53e58119712758f (DEFAULT_TABLE table=rebalance_test_table_2 [id=632b93f2c4174b0080e62682cf1200b4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:36.670058 5883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 098804415c3b4bafa53e58119712758f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:36.671655 5885 tablet_service.cc:1467] Processing CreateTablet for tablet de3d8ce603a2444e8372626dd6b39b6c (DEFAULT_TABLE table=rebalance_test_table_2 [id=632b93f2c4174b0080e62682cf1200b4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:36.672048 5885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3d8ce603a2444e8372626dd6b39b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:36.672919 5621 tablet_service.cc:1467] Processing CreateTablet for tablet 16205fdbbb994e9f840c77dfd83dad1f (DEFAULT_TABLE table=rebalance_test_table_2 [id=632b93f2c4174b0080e62682cf1200b4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:36.673318 5621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16205fdbbb994e9f840c77dfd83dad1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:36.676007 5752 tablet_service.cc:1467] Processing CreateTablet for tablet 16205fdbbb994e9f840c77dfd83dad1f (DEFAULT_TABLE table=rebalance_test_table_2 [id=632b93f2c4174b0080e62682cf1200b4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:36.676498 5884 tablet_service.cc:1467] Processing CreateTablet for tablet 16205fdbbb994e9f840c77dfd83dad1f (DEFAULT_TABLE table=rebalance_test_table_2 [id=632b93f2c4174b0080e62682cf1200b4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:36.676841 5884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16205fdbbb994e9f840c77dfd83dad1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:36.676832 5752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16205fdbbb994e9f840c77dfd83dad1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:36.676198 5754 tablet_service.cc:1467] Processing CreateTablet for tablet de3d8ce603a2444e8372626dd6b39b6c (DEFAULT_TABLE table=rebalance_test_table_2 [id=632b93f2c4174b0080e62682cf1200b4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:36.680459 5754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3d8ce603a2444e8372626dd6b39b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:36.680511 5622 tablet_service.cc:1467] Processing CreateTablet for tablet de3d8ce603a2444e8372626dd6b39b6c (DEFAULT_TABLE table=rebalance_test_table_2 [id=632b93f2c4174b0080e62682cf1200b4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:36.680858 5622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3d8ce603a2444e8372626dd6b39b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:36.685974 6544 tablet_bootstrap.cc:492] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9: Bootstrap starting.
I20240503 14:14:36.689188 6544 tablet_bootstrap.cc:654] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:36.696781 6543 tablet_bootstrap.cc:492] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33: Bootstrap starting.
I20240503 14:14:36.697613 6545 tablet_bootstrap.cc:492] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f: Bootstrap starting.
I20240503 14:14:36.699153 6543 tablet_bootstrap.cc:654] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:36.699729 6545 tablet_bootstrap.cc:654] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:36.701946 6544 tablet_bootstrap.cc:492] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9: No bootstrap required, opened a new log
I20240503 14:14:36.702069 6543 tablet_bootstrap.cc:492] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33: No bootstrap required, opened a new log
I20240503 14:14:36.702090 6544 ts_tablet_manager.cc:1367] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20240503 14:14:36.702220 6545 tablet_bootstrap.cc:492] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f: No bootstrap required, opened a new log
I20240503 14:14:36.702226 6543 ts_tablet_manager.cc:1367] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240503 14:14:36.702363 6545 ts_tablet_manager.cc:1367] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:14:36.702868 6544 raft_consensus.cc:348] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.703079 6544 raft_consensus.cc:374] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:36.703029 6543 raft_consensus.cc:348] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.703171 6544 raft_consensus.cc:729] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a749bf14364f22a4db0f7cdfa33ce9, State: Initialized, Role: FOLLOWER
I20240503 14:14:36.703227 6543 raft_consensus.cc:374] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:36.703325 6543 raft_consensus.cc:729] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5237ac15b5c4a6c88a8a07f3d44cf33, State: Initialized, Role: FOLLOWER
I20240503 14:14:36.703297 6545 raft_consensus.cc:348] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.703449 6544 consensus_queue.cc:260] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.703706 6543 consensus_queue.cc:260] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.704180 6544 ts_tablet_manager.cc:1398] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:14:36.704296 6543 ts_tablet_manager.cc:1398] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:14:36.704510 6544 tablet_bootstrap.cc:492] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9: Bootstrap starting.
I20240503 14:14:36.705128 6543 tablet_bootstrap.cc:492] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33: Bootstrap starting.
I20240503 14:14:36.706240 6544 tablet_bootstrap.cc:654] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:36.708300 6544 tablet_bootstrap.cc:492] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9: No bootstrap required, opened a new log
I20240503 14:14:36.708891 6543 tablet_bootstrap.cc:654] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:36.710850 6544 ts_tablet_manager.cc:1367] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240503 14:14:36.711469 6544 raft_consensus.cc:348] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.711664 6544 raft_consensus.cc:374] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:36.711740 6544 raft_consensus.cc:729] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a749bf14364f22a4db0f7cdfa33ce9, State: Initialized, Role: FOLLOWER
I20240503 14:14:36.711961 6544 consensus_queue.cc:260] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.703511 6545 raft_consensus.cc:374] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:36.712515 6545 raft_consensus.cc:729] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5361a07e89ad47298aabe717a4f2867f, State: Initialized, Role: FOLLOWER
I20240503 14:14:36.712565 6544 ts_tablet_manager.cc:1398] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:14:36.712913 6544 tablet_bootstrap.cc:492] T 16205fdbbb994e9f840c77dfd83dad1f P a8a749bf14364f22a4db0f7cdfa33ce9: Bootstrap starting.
I20240503 14:14:36.713193 6545 consensus_queue.cc:260] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.714633 6545 ts_tablet_manager.cc:1398] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 14:14:36.715433 6544 tablet_bootstrap.cc:654] T 16205fdbbb994e9f840c77dfd83dad1f P a8a749bf14364f22a4db0f7cdfa33ce9: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:36.715579 6545 tablet_bootstrap.cc:492] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: Bootstrap starting.
I20240503 14:14:36.717767 6545 tablet_bootstrap.cc:654] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:36.721813 6543 tablet_bootstrap.cc:492] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33: No bootstrap required, opened a new log
I20240503 14:14:36.721992 6543 ts_tablet_manager.cc:1367] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.009s
I20240503 14:14:36.722704 6544 tablet_bootstrap.cc:492] T 16205fdbbb994e9f840c77dfd83dad1f P a8a749bf14364f22a4db0f7cdfa33ce9: No bootstrap required, opened a new log
I20240503 14:14:36.722672 6543 raft_consensus.cc:348] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.722832 6544 ts_tablet_manager.cc:1367] T 16205fdbbb994e9f840c77dfd83dad1f P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20240503 14:14:36.722885 6543 raft_consensus.cc:374] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:36.722968 6543 raft_consensus.cc:729] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5237ac15b5c4a6c88a8a07f3d44cf33, State: Initialized, Role: FOLLOWER
I20240503 14:14:36.723203 6543 consensus_queue.cc:260] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.723489 6544 raft_consensus.cc:348] T 16205fdbbb994e9f840c77dfd83dad1f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.723690 6543 ts_tablet_manager.cc:1398] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:36.723699 6544 raft_consensus.cc:374] T 16205fdbbb994e9f840c77dfd83dad1f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:36.723788 6544 raft_consensus.cc:729] T 16205fdbbb994e9f840c77dfd83dad1f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a749bf14364f22a4db0f7cdfa33ce9, State: Initialized, Role: FOLLOWER
I20240503 14:14:36.724086 6543 tablet_bootstrap.cc:492] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33: Bootstrap starting.
I20240503 14:14:36.724103 6544 consensus_queue.cc:260] T 16205fdbbb994e9f840c77dfd83dad1f P a8a749bf14364f22a4db0f7cdfa33ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.724654 6545 tablet_bootstrap.cc:492] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: No bootstrap required, opened a new log
I20240503 14:14:36.724886 6545 ts_tablet_manager.cc:1367] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240503 14:14:36.725373 6544 ts_tablet_manager.cc:1398] T 16205fdbbb994e9f840c77dfd83dad1f P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:14:36.725618 6375 raft_consensus.cc:482] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:36.725769 6543 tablet_bootstrap.cc:654] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:36.725661 6545 raft_consensus.cc:348] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.725798 6375 raft_consensus.cc:504] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.725945 6545 raft_consensus.cc:374] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:36.726042 6545 raft_consensus.cc:729] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5361a07e89ad47298aabe717a4f2867f, State: Initialized, Role: FOLLOWER
I20240503 14:14:36.726342 6545 consensus_queue.cc:260] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.726459 6375 leader_election.cc:288] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157), 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071)
I20240503 14:14:36.726966 5773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3d8ce603a2444e8372626dd6b39b6c" candidate_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5361a07e89ad47298aabe717a4f2867f" is_pre_election: true
I20240503 14:14:36.727105 6545 ts_tablet_manager.cc:1398] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:14:36.727500 6545 tablet_bootstrap.cc:492] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f: Bootstrap starting.
W20240503 14:14:36.727818 5840 leader_election.cc:341] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:14:36.729159 6545 tablet_bootstrap.cc:654] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:36.729300 6543 tablet_bootstrap.cc:492] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33: No bootstrap required, opened a new log
I20240503 14:14:36.729496 6543 ts_tablet_manager.cc:1367] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240503 14:14:36.730264 6543 raft_consensus.cc:348] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.730528 6543 raft_consensus.cc:374] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:36.730731 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3d8ce603a2444e8372626dd6b39b6c" candidate_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" is_pre_election: true
I20240503 14:14:36.730871 6543 raft_consensus.cc:729] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5237ac15b5c4a6c88a8a07f3d44cf33, State: Initialized, Role: FOLLOWER
I20240503 14:14:36.731119 6543 consensus_queue.cc:260] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.731387 5641 raft_consensus.cc:2454] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8a749bf14364f22a4db0f7cdfa33ce9 in term 0.
I20240503 14:14:36.731818 5838 leader_election.cc:302] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a8a749bf14364f22a4db0f7cdfa33ce9, e5237ac15b5c4a6c88a8a07f3d44cf33; no voters: 5361a07e89ad47298aabe717a4f2867f
I20240503 14:14:36.732095 6543 ts_tablet_manager.cc:1398] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:14:36.732208 6375 raft_consensus.cc:2781] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:36.732309 6375 raft_consensus.cc:482] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:36.732419 6375 raft_consensus.cc:3037] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:36.732636 6545 tablet_bootstrap.cc:492] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f: No bootstrap required, opened a new log
I20240503 14:14:36.734479 6545 ts_tablet_manager.cc:1367] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 14:14:36.735301 6545 raft_consensus.cc:348] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.735482 6545 raft_consensus.cc:374] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:36.735585 6545 raft_consensus.cc:729] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5361a07e89ad47298aabe717a4f2867f, State: Initialized, Role: FOLLOWER
I20240503 14:14:36.735791 6545 consensus_queue.cc:260] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.735883 6375 raft_consensus.cc:504] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.736269 6545 ts_tablet_manager.cc:1398] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:36.736555 6375 leader_election.cc:288] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [CANDIDATE]: Term 1 election: Requested vote from peers e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157), 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071)
I20240503 14:14:36.736856 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3d8ce603a2444e8372626dd6b39b6c" candidate_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33"
I20240503 14:14:36.736958 5773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3d8ce603a2444e8372626dd6b39b6c" candidate_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5361a07e89ad47298aabe717a4f2867f"
I20240503 14:14:36.737040 5641 raft_consensus.cc:3037] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:36.737174 5773 raft_consensus.cc:3037] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:36.739503 5773 raft_consensus.cc:2454] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8a749bf14364f22a4db0f7cdfa33ce9 in term 1.
I20240503 14:14:36.739830 5641 raft_consensus.cc:2454] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8a749bf14364f22a4db0f7cdfa33ce9 in term 1.
I20240503 14:14:36.739859 5840 leader_election.cc:302] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5361a07e89ad47298aabe717a4f2867f, a8a749bf14364f22a4db0f7cdfa33ce9; no voters:
I20240503 14:14:36.740134 6375 raft_consensus.cc:2781] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:36.740638 6375 raft_consensus.cc:686] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Becoming Leader. State: Replica: a8a749bf14364f22a4db0f7cdfa33ce9, State: Running, Role: LEADER
I20240503 14:14:36.740960 6375 consensus_queue.cc:237] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.743299 5498 catalog_manager.cc:5484] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67). New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:36.771402 6375 raft_consensus.cc:482] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:36.771598 6375 raft_consensus.cc:504] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.772307 6375 leader_election.cc:288] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157), 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071)
I20240503 14:14:36.772727 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098804415c3b4bafa53e58119712758f" candidate_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" is_pre_election: true
I20240503 14:14:36.772837 5773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098804415c3b4bafa53e58119712758f" candidate_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5361a07e89ad47298aabe717a4f2867f" is_pre_election: true
I20240503 14:14:36.773010 5641 raft_consensus.cc:2454] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8a749bf14364f22a4db0f7cdfa33ce9 in term 0.
I20240503 14:14:36.773056 5773 raft_consensus.cc:2454] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8a749bf14364f22a4db0f7cdfa33ce9 in term 0.
I20240503 14:14:36.773452 5838 leader_election.cc:302] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5361a07e89ad47298aabe717a4f2867f, a8a749bf14364f22a4db0f7cdfa33ce9, e5237ac15b5c4a6c88a8a07f3d44cf33; no voters:
I20240503 14:14:36.773768 6375 raft_consensus.cc:2781] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:36.773878 6375 raft_consensus.cc:482] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:36.773967 6375 raft_consensus.cc:3037] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:36.775848 6375 raft_consensus.cc:504] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.776417 6375 leader_election.cc:288] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [CANDIDATE]: Term 1 election: Requested vote from peers e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157), 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071)
I20240503 14:14:36.776878 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098804415c3b4bafa53e58119712758f" candidate_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33"
I20240503 14:14:36.776899 5773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098804415c3b4bafa53e58119712758f" candidate_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5361a07e89ad47298aabe717a4f2867f"
I20240503 14:14:36.777051 5641 raft_consensus.cc:3037] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:36.777071 5773 raft_consensus.cc:3037] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:36.778875 5773 raft_consensus.cc:2454] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8a749bf14364f22a4db0f7cdfa33ce9 in term 1.
I20240503 14:14:36.778880 5641 raft_consensus.cc:2454] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8a749bf14364f22a4db0f7cdfa33ce9 in term 1.
I20240503 14:14:36.779260 5838 leader_election.cc:302] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8a749bf14364f22a4db0f7cdfa33ce9, e5237ac15b5c4a6c88a8a07f3d44cf33; no voters:
I20240503 14:14:36.779547 6375 raft_consensus.cc:2781] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:36.779691 6375 raft_consensus.cc:686] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Becoming Leader. State: Replica: a8a749bf14364f22a4db0f7cdfa33ce9, State: Running, Role: LEADER
I20240503 14:14:36.779968 6375 consensus_queue.cc:237] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.782495 5498 catalog_manager.cc:5484] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67). New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:36.793444 6476 raft_consensus.cc:482] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:36.793601 6476 raft_consensus.cc:504] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.794237 6476 leader_election.cc:288] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157), a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
I20240503 14:14:36.794621 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16205fdbbb994e9f840c77dfd83dad1f" candidate_uuid: "5361a07e89ad47298aabe717a4f2867f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" is_pre_election: true
I20240503 14:14:36.794672 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16205fdbbb994e9f840c77dfd83dad1f" candidate_uuid: "5361a07e89ad47298aabe717a4f2867f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" is_pre_election: true
I20240503 14:14:36.794811 5641 raft_consensus.cc:2454] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5361a07e89ad47298aabe717a4f2867f in term 0.
I20240503 14:14:36.794900 5905 raft_consensus.cc:2454] T 16205fdbbb994e9f840c77dfd83dad1f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5361a07e89ad47298aabe717a4f2867f in term 0.
I20240503 14:14:36.795198 5707 leader_election.cc:302] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5361a07e89ad47298aabe717a4f2867f, e5237ac15b5c4a6c88a8a07f3d44cf33; no voters:
I20240503 14:14:36.795442 6476 raft_consensus.cc:2781] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:36.795557 6476 raft_consensus.cc:482] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:36.795658 6476 raft_consensus.cc:3037] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:36.797593 6476 raft_consensus.cc:504] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.798094 6476 leader_election.cc:288] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [CANDIDATE]: Term 1 election: Requested vote from peers e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157), a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
I20240503 14:14:36.798533 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16205fdbbb994e9f840c77dfd83dad1f" candidate_uuid: "5361a07e89ad47298aabe717a4f2867f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33"
I20240503 14:14:36.798620 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16205fdbbb994e9f840c77dfd83dad1f" candidate_uuid: "5361a07e89ad47298aabe717a4f2867f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9"
I20240503 14:14:36.798691 5641 raft_consensus.cc:3037] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:36.798823 5905 raft_consensus.cc:3037] T 16205fdbbb994e9f840c77dfd83dad1f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:36.800556 5641 raft_consensus.cc:2454] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5361a07e89ad47298aabe717a4f2867f in term 1.
I20240503 14:14:36.800696 5905 raft_consensus.cc:2454] T 16205fdbbb994e9f840c77dfd83dad1f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5361a07e89ad47298aabe717a4f2867f in term 1.
I20240503 14:14:36.800935 5707 leader_election.cc:302] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5361a07e89ad47298aabe717a4f2867f, e5237ac15b5c4a6c88a8a07f3d44cf33; no voters:
I20240503 14:14:36.801216 6476 raft_consensus.cc:2781] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:36.801576 6476 raft_consensus.cc:686] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Becoming Leader. State: Replica: 5361a07e89ad47298aabe717a4f2867f, State: Running, Role: LEADER
I20240503 14:14:36.801879 6476 consensus_queue.cc:237] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } }
I20240503 14:14:36.804477 5498 catalog_manager.cc:5484] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5361a07e89ad47298aabe717a4f2867f (127.22.242.66). New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:37.138439 6375 consensus_queue.cc:1035] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:37.144332 6375 consensus_queue.cc:1035] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:14:37.166923 6561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:37.167280 6561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:37.174831 6561 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 14:14:37.228287 6375 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:37.238040 6375 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:37.244719 6476 consensus_queue.cc:1035] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:37.278350 6476 consensus_queue.cc:1035] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 299.111 ms
time per row: 0.149555 ms
I20240503 14:14:37.624994 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.69:42145
--local_ip_for_outbound_sockets=127.22.242.69
--tserver_master_addrs=127.22.242.126:41785
--webserver_port=41551
--webserver_interface=127.22.242.69
--builtin_ntp_servers=127.22.242.84:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:14:37.977519 6621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:37.977813 6621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:37.977941 6621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:37.987165 6621 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:37.987378 6621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:37.987627 6621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.69
I20240503 14:14:37.998339 6621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.69:42145
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.242.69
--webserver_port=41551
--tserver_master_addrs=127.22.242.126:41785
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.69
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:37.999768 6621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:38.001659 6621 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:38.013990 6626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:38.014288 6627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:38.014288 6629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:38.015260 6621 server_base.cc:1034] running on GCE node
I20240503 14:14:38.015956 6621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:38.016997 6621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:38.018201 6621 hybrid_clock.cc:648] HybridClock initialized: now 1714745678018142 us; error 56 us; skew 500 ppm
I20240503 14:14:38.018519 6621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:38.020807 6621 webserver.cc:460] Webserver started at http://127.22.242.69:41551/ using document root <none> and password file <none>
I20240503 14:14:38.021559 6621 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:38.021698 6621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:38.027245 6621 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:14:38.030141 6635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:38.031500 6621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:38.031653 6621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a24048f65ee54156b569d4bf1f738368"
format_stamp: "Formatted at 2024-05-03 14:14:27 on dist-test-slave-r3pd"
I20240503 14:14:38.032532 6621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:38.057669 6621 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:38.058514 6621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:38.058800 6621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:38.059825 6621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:38.061435 6621 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:38.061506 6621 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:38.061604 6621 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:38.061647 6621 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:38.093663 6621 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.69:42145
I20240503 14:14:38.093699 6747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.69:42145 every 8 connection(s)
I20240503 14:14:38.095075 6621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:14:38.103570 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 6621
I20240503 14:14:38.104408 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.70:43863
--local_ip_for_outbound_sockets=127.22.242.70
--tserver_master_addrs=127.22.242.126:41785
--webserver_port=40941
--webserver_interface=127.22.242.70
--builtin_ntp_servers=127.22.242.84:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:14:38.107911 6748 heartbeater.cc:347] Connected to a master server at 127.22.242.126:41785
I20240503 14:14:38.108314 6748 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:38.109373 6748 heartbeater.cc:511] Master 127.22.242.126:41785 requested a full tablet report, sending...
I20240503 14:14:38.111222 5494 ts_manager.cc:194] Re-registered known tserver with Master: a24048f65ee54156b569d4bf1f738368 (127.22.242.69:42145)
I20240503 14:14:38.112094 5494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.69:44075
W20240503 14:14:38.448837 6752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:38.449139 6752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:38.449226 6752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:38.457289 6752 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:38.457419 6752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:38.457589 6752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.70
I20240503 14:14:38.466919 6752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.70:43863
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.242.70
--webserver_port=40941
--tserver_master_addrs=127.22.242.126:41785
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.70
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:38.468286 6752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:38.469939 6752 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:38.482434 6758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:38.482677 6757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:38.483183 6760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:38.483841 6752 server_base.cc:1034] running on GCE node
I20240503 14:14:38.484634 6752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:38.485735 6752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:38.486953 6752 hybrid_clock.cc:648] HybridClock initialized: now 1714745678486907 us; error 40 us; skew 500 ppm
I20240503 14:14:38.487255 6752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:38.494011 6752 webserver.cc:460] Webserver started at http://127.22.242.70:40941/ using document root <none> and password file <none>
I20240503 14:14:38.494757 6752 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:38.494890 6752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:38.499979 6752 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:38.502707 6766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:38.504098 6752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:38.504313 6752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "7bb84965b41b41fa9be23175c82e0a3a"
format_stamp: "Formatted at 2024-05-03 14:14:28 on dist-test-slave-r3pd"
I20240503 14:14:38.505259 6752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:38.526937 6752 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:38.527707 6752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:38.527964 6752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:38.528911 6752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:38.530403 6752 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:38.530474 6752 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:38.530565 6752 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:38.530616 6752 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:38.562510 6752 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.70:43863
I20240503 14:14:38.562566 6878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.70:43863 every 8 connection(s)
I20240503 14:14:38.564088 6752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:14:38.565557 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 6752
I20240503 14:14:38.566205 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.68:42505
--local_ip_for_outbound_sockets=127.22.242.68
--tserver_master_addrs=127.22.242.126:41785
--webserver_port=42357
--webserver_interface=127.22.242.68
--builtin_ntp_servers=127.22.242.84:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:14:38.577296 6879 heartbeater.cc:347] Connected to a master server at 127.22.242.126:41785
I20240503 14:14:38.577612 6879 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:38.578505 6879 heartbeater.cc:511] Master 127.22.242.126:41785 requested a full tablet report, sending...
I20240503 14:14:38.580106 5494 ts_manager.cc:194] Re-registered known tserver with Master: 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70:43863)
I20240503 14:14:38.581238 5494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.70:56181
W20240503 14:14:38.903309 6882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:38.903596 6882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:38.903682 6882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:38.911274 6882 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:38.911415 6882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:38.911568 6882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.68
I20240503 14:14:38.921073 6882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.68:42505
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.242.68
--webserver_port=42357
--tserver_master_addrs=127.22.242.126:41785
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.68
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:38.922650 6882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:38.924984 6882 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:38.938253 6887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:38.938429 6888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:38.938711 6890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:38.939517 6882 server_base.cc:1034] running on GCE node
I20240503 14:14:38.940079 6882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:38.941088 6882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:38.942320 6882 hybrid_clock.cc:648] HybridClock initialized: now 1714745678942272 us; error 44 us; skew 500 ppm
I20240503 14:14:38.942632 6882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:38.945238 6882 webserver.cc:460] Webserver started at http://127.22.242.68:42357/ using document root <none> and password file <none>
I20240503 14:14:38.945984 6882 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:38.946116 6882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:38.951174 6882 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20240503 14:14:38.953872 6896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:38.955330 6882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:14:38.955497 6882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "da201109e13f4b36b3fba7477454a4c6"
format_stamp: "Formatted at 2024-05-03 14:14:27 on dist-test-slave-r3pd"
I20240503 14:14:38.956311 6882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:38.970225 6882 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:38.971005 6882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:38.971269 6882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:38.972127 6882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:38.973628 6882 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:38.973711 6882 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:38.973831 6882 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:38.973888 6882 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:39.004297 6882 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.68:42505
I20240503 14:14:39.004345 7008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.68:42505 every 8 connection(s)
I20240503 14:14:39.005774 6882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:14:39.006273 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 6882
I20240503 14:14:39.021827 7009 heartbeater.cc:347] Connected to a master server at 127.22.242.126:41785
I20240503 14:14:39.022189 7009 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:39.023084 7009 heartbeater.cc:511] Master 127.22.242.126:41785 requested a full tablet report, sending...
I20240503 14:14:39.024856 5494 ts_manager.cc:194] Re-registered known tserver with Master: da201109e13f4b36b3fba7477454a4c6 (127.22.242.68:42505)
I20240503 14:14:39.026038 5494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.68:58245
I20240503 14:14:39.115689 6748 heartbeater.cc:503] Master 127.22.242.126:41785 was elected leader, sending a full tablet report...
I20240503 14:14:39.489800 6814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:39.491865 5752 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:39.526393 5884 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:39.538143 6683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:39.546993 6944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:39.583833 6879 heartbeater.cc:503] Master 127.22.242.126:41785 was elected leader, sending a full tablet report...
I20240503 14:14:39.587774 5620 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:39.724032 5752 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:39.733299 6683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:39.742892 6814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:39.759560 5884 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:39.770712 6944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:39.787771 5620 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:39.937042 5752 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:39.938689 6814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:39.953146 6683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:40.000800 5884 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:40.028649 7009 heartbeater.cc:503] Master 127.22.242.126:41785 was elected leader, sending a full tablet report...
I20240503 14:14:40.030441 6944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:40.030522 5620 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:40.166838 6683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:40.170828 5752 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:40.173687 5884 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:40.174415 6814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:40.183100 6944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:40.205255 5620 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:40.275848 5904 consensus_queue.cc:237] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } }
I20240503 14:14:40.278709 5773 raft_consensus.cc:1261] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.279012 5642 raft_consensus.cc:1261] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.279249 6375 consensus_queue.cc:1035] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.279950 7155 consensus_queue.cc:1035] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.288403 6375 raft_consensus.cc:2932] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } } }
W20240503 14:14:40.290544 5839 consensus_peers.cc:485] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 -> Peer 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70:43863): Couldn't send request to peer 7bb84965b41b41fa9be23175c82e0a3a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de3d8ce603a2444e8372626dd6b39b6c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:40.294010 5642 raft_consensus.cc:2932] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } } }
I20240503 14:14:40.295960 5773 raft_consensus.cc:2932] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } } }
I20240503 14:14:40.296659 5495 catalog_manager.cc:5484] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 reported cstate change: config changed from index -1 to 4, NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:40.315682 5773 consensus_queue.cc:237] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } }
I20240503 14:14:40.318035 5642 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.319949 6476 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.321869 5904 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.322741 6476 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:14:40.329995 5710 consensus_peers.cc:485] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f -> Peer a24048f65ee54156b569d4bf1f738368 (127.22.242.69:42145): Couldn't send request to peer a24048f65ee54156b569d4bf1f738368. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 614d4da74aa547a28013c97f5a1079fd. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:40.336453 6476 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } }
I20240503 14:14:40.337903 5904 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } }
I20240503 14:14:40.338675 5642 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } }
I20240503 14:14:40.342864 5494 catalog_manager.cc:5484] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f reported cstate change: config changed from index -1 to 4, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:40.354738 5904 consensus_queue.cc:237] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } }
I20240503 14:14:40.357189 5773 raft_consensus.cc:1261] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.357214 5642 raft_consensus.cc:1261] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.358108 6375 consensus_queue.cc:1035] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.358387 7155 consensus_queue.cc:1035] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.361425 7160 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } } }
I20240503 14:14:40.363425 5642 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } } }
W20240503 14:14:40.363909 5838 consensus_peers.cc:485] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 -> Peer da201109e13f4b36b3fba7477454a4c6 (127.22.242.68:42505): Couldn't send request to peer da201109e13f4b36b3fba7477454a4c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6b542b55eb9477ab472b3a4005e6c63. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:40.364775 5773 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } } }
I20240503 14:14:40.366824 5495 catalog_manager.cc:5484] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 reported cstate change: config changed from index -1 to 4, NON_VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) added. New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:40.385385 5642 consensus_queue.cc:237] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } }
I20240503 14:14:40.388787 5773 raft_consensus.cc:1261] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.388862 5904 raft_consensus.cc:1261] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.389566 7190 consensus_queue.cc:1035] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.390031 6386 consensus_queue.cc:1035] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.393218 6386 raft_consensus.cc:2932] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } } }
I20240503 14:14:40.393945 5904 raft_consensus.cc:2932] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } } }
I20240503 14:14:40.394305 5773 raft_consensus.cc:2932] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } } }
W20240503 14:14:40.396977 5576 consensus_peers.cc:485] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 -> Peer 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70:43863): Couldn't send request to peer 7bb84965b41b41fa9be23175c82e0a3a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b8188059098481e9dd4b46858bda06c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:40.399966 5494 catalog_manager.cc:5484] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9 reported cstate change: config changed from index -1 to 4, NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } } }
I20240503 14:14:40.419919 5642 consensus_queue.cc:237] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } }
I20240503 14:14:40.422528 5773 raft_consensus.cc:1261] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.423137 5904 raft_consensus.cc:1261] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.423187 7195 consensus_queue.cc:1035] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.423671 6386 consensus_queue.cc:1035] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.426159 7195 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } }
I20240503 14:14:40.426796 5773 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } }
W20240503 14:14:40.429046 5578 consensus_peers.cc:485] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 -> Peer a24048f65ee54156b569d4bf1f738368 (127.22.242.69:42145): Couldn't send request to peer a24048f65ee54156b569d4bf1f738368. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb339ed8f9a141caa62cc5588d07fee1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:40.432469 5494 catalog_manager.cc:5484] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f reported cstate change: config changed from index -1 to 4, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } }
I20240503 14:14:40.433717 5904 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } }
I20240503 14:14:40.457104 5773 consensus_queue.cc:237] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } }
I20240503 14:14:40.460309 5904 raft_consensus.cc:1261] T 16205fdbbb994e9f840c77dfd83dad1f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.460354 5642 raft_consensus.cc:1261] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.460922 7172 consensus_queue.cc:1035] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.461550 7172 consensus_queue.cc:1035] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:14:40.468849 5707 consensus_peers.cc:485] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f -> Peer da201109e13f4b36b3fba7477454a4c6 (127.22.242.68:42505): Couldn't send request to peer da201109e13f4b36b3fba7477454a4c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16205fdbbb994e9f840c77dfd83dad1f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:40.473795 7171 raft_consensus.cc:2932] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } } }
I20240503 14:14:40.474761 5641 raft_consensus.cc:2932] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } } }
I20240503 14:14:40.474979 5904 raft_consensus.cc:2932] T 16205fdbbb994e9f840c77dfd83dad1f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } } }
I20240503 14:14:40.480544 5494 catalog_manager.cc:5484] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33 reported cstate change: config changed from index -1 to 4, NON_VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) added. New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } } }
I20240503 14:14:40.499507 5904 consensus_queue.cc:237] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } }
I20240503 14:14:40.501698 5641 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.501763 5773 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:40.501983 5839 consensus_peers.cc:485] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 -> Peer 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70:43863): Couldn't send request to peer 7bb84965b41b41fa9be23175c82e0a3a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 098804415c3b4bafa53e58119712758f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:40.502367 7155 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.502614 7160 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.505215 6375 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } } }
I20240503 14:14:40.505791 5773 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } } }
I20240503 14:14:40.510058 5495 catalog_manager.cc:5484] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f reported cstate change: config changed from index -1 to 4, NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } } }
I20240503 14:14:40.511890 5641 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } } }
I20240503 14:14:40.523378 5773 consensus_queue.cc:237] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } }
I20240503 14:14:40.525842 5641 raft_consensus.cc:1261] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.525882 5904 raft_consensus.cc:1261] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:40.526000 5710 consensus_peers.cc:485] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f -> Peer a24048f65ee54156b569d4bf1f738368 (127.22.242.69:42145): Couldn't send request to peer a24048f65ee54156b569d4bf1f738368. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52983271755542f090bb26d0675ec84e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:40.526338 7171 consensus_queue.cc:1035] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.526876 7171 consensus_queue.cc:1035] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.528932 7171 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } }
I20240503 14:14:40.529525 5641 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } }
I20240503 14:14:40.533244 5495 catalog_manager.cc:5484] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f reported cstate change: config changed from index -1 to 4, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:40.533272 5904 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } }
I20240503 14:14:40.747596 7270 ts_tablet_manager.cc:902] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a: Initiating tablet copy from peer a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
I20240503 14:14:40.749332 7270 tablet_copy_client.cc:308] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44189
I20240503 14:14:40.755548 5925 tablet_copy_service.cc:140] P a8a749bf14364f22a4db0f7cdfa33ce9: Received BeginTabletCopySession request for tablet de3d8ce603a2444e8372626dd6b39b6c from peer 7bb84965b41b41fa9be23175c82e0a3a ({username='slave'} at 127.22.242.70:37583)
I20240503 14:14:40.755777 5925 tablet_copy_service.cc:161] P a8a749bf14364f22a4db0f7cdfa33ce9: Beginning new tablet copy session on tablet de3d8ce603a2444e8372626dd6b39b6c from peer 7bb84965b41b41fa9be23175c82e0a3a at {username='slave'} at 127.22.242.70:37583: session id = 7bb84965b41b41fa9be23175c82e0a3a-de3d8ce603a2444e8372626dd6b39b6c
I20240503 14:14:40.758517 5925 tablet_copy_source_session.cc:205] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:40.762972 7270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3d8ce603a2444e8372626dd6b39b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:40.772507 7270 tablet_copy_client.cc:783] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:40.772768 7270 tablet_copy_client.cc:647] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:40.775125 7270 tablet_copy_client.cc:523] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:40.778045 7270 tablet_bootstrap.cc:492] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a: Bootstrap starting.
I20240503 14:14:40.787168 7270 log.cc:824] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:40.816812 7282 ts_tablet_manager.cc:902] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368: Initiating tablet copy from peer 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071)
I20240503 14:14:40.824416 7270 tablet_bootstrap.cc:492] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:40.825197 7270 tablet_bootstrap.cc:492] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a: Bootstrap complete.
I20240503 14:14:40.825834 7270 ts_tablet_manager.cc:1367] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a: Time spent bootstrapping tablet: real 0.048s user 0.045s sys 0.004s
I20240503 14:14:40.833048 7282 tablet_copy_client.cc:308] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:41071
I20240503 14:14:40.834429 7270 raft_consensus.cc:348] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } }
I20240503 14:14:40.835021 7270 raft_consensus.cc:729] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7bb84965b41b41fa9be23175c82e0a3a, State: Initialized, Role: LEARNER
I20240503 14:14:40.835656 7270 consensus_queue.cc:260] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } }
I20240503 14:14:40.842053 7270 ts_tablet_manager.cc:1398] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20240503 14:14:40.843338 5925 tablet_copy_service.cc:342] P a8a749bf14364f22a4db0f7cdfa33ce9: Request end of tablet copy session 7bb84965b41b41fa9be23175c82e0a3a-de3d8ce603a2444e8372626dd6b39b6c received from {username='slave'} at 127.22.242.70:37583
I20240503 14:14:40.843539 5925 tablet_copy_service.cc:433] P a8a749bf14364f22a4db0f7cdfa33ce9: ending tablet copy session 7bb84965b41b41fa9be23175c82e0a3a-de3d8ce603a2444e8372626dd6b39b6c on tablet de3d8ce603a2444e8372626dd6b39b6c with peer 7bb84965b41b41fa9be23175c82e0a3a
I20240503 14:14:40.847126 5794 tablet_copy_service.cc:140] P 5361a07e89ad47298aabe717a4f2867f: Received BeginTabletCopySession request for tablet 614d4da74aa547a28013c97f5a1079fd from peer a24048f65ee54156b569d4bf1f738368 ({username='slave'} at 127.22.242.69:38851)
I20240503 14:14:40.847420 5794 tablet_copy_service.cc:161] P 5361a07e89ad47298aabe717a4f2867f: Beginning new tablet copy session on tablet 614d4da74aa547a28013c97f5a1079fd from peer a24048f65ee54156b569d4bf1f738368 at {username='slave'} at 127.22.242.69:38851: session id = a24048f65ee54156b569d4bf1f738368-614d4da74aa547a28013c97f5a1079fd
I20240503 14:14:40.847827 7295 ts_tablet_manager.cc:902] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6: Initiating tablet copy from peer a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
I20240503 14:14:40.851161 5794 tablet_copy_source_session.cc:205] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:40.854933 7295 tablet_copy_client.cc:308] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44189
I20240503 14:14:40.858307 7282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 614d4da74aa547a28013c97f5a1079fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:40.866097 5925 tablet_copy_service.cc:140] P a8a749bf14364f22a4db0f7cdfa33ce9: Received BeginTabletCopySession request for tablet f6b542b55eb9477ab472b3a4005e6c63 from peer da201109e13f4b36b3fba7477454a4c6 ({username='slave'} at 127.22.242.68:54621)
I20240503 14:14:40.866266 5925 tablet_copy_service.cc:161] P a8a749bf14364f22a4db0f7cdfa33ce9: Beginning new tablet copy session on tablet f6b542b55eb9477ab472b3a4005e6c63 from peer da201109e13f4b36b3fba7477454a4c6 at {username='slave'} at 127.22.242.68:54621: session id = da201109e13f4b36b3fba7477454a4c6-f6b542b55eb9477ab472b3a4005e6c63
I20240503 14:14:40.869345 5925 tablet_copy_source_session.cc:205] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:40.871548 7282 tablet_copy_client.cc:783] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:40.871851 7282 tablet_copy_client.cc:647] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:40.874307 7295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b542b55eb9477ab472b3a4005e6c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:40.874482 7282 tablet_copy_client.cc:523] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:40.878021 7282 tablet_bootstrap.cc:492] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368: Bootstrap starting.
I20240503 14:14:40.888291 7295 tablet_copy_client.cc:783] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:40.888706 7295 tablet_copy_client.cc:647] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:40.891084 7282 log.cc:824] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:40.893947 7295 tablet_copy_client.cc:523] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:40.899461 7295 tablet_bootstrap.cc:492] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6: Bootstrap starting.
I20240503 14:14:40.912633 7295 log.cc:824] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:40.945341 7282 tablet_bootstrap.cc:492] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:40.945963 7282 tablet_bootstrap.cc:492] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368: Bootstrap complete.
I20240503 14:14:40.946460 7282 ts_tablet_manager.cc:1367] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368: Time spent bootstrapping tablet: real 0.069s user 0.053s sys 0.011s
I20240503 14:14:40.953480 7282 raft_consensus.cc:348] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } }
I20240503 14:14:40.954030 7282 raft_consensus.cc:729] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a24048f65ee54156b569d4bf1f738368, State: Initialized, Role: LEARNER
I20240503 14:14:40.954634 7282 consensus_queue.cc:260] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } }
I20240503 14:14:40.969039 7295 tablet_bootstrap.cc:492] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:40.969611 7295 tablet_bootstrap.cc:492] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6: Bootstrap complete.
I20240503 14:14:40.970820 7295 ts_tablet_manager.cc:1367] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6: Time spent bootstrapping tablet: real 0.072s user 0.060s sys 0.008s
I20240503 14:14:40.971510 7270 ts_tablet_manager.cc:902] T 7b8188059098481e9dd4b46858bda06c P 7bb84965b41b41fa9be23175c82e0a3a: Initiating tablet copy from peer e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157)
I20240503 14:14:40.973543 7282 ts_tablet_manager.cc:1398] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368: Time spent starting tablet: real 0.027s user 0.012s sys 0.006s
I20240503 14:14:40.976668 5794 tablet_copy_service.cc:342] P 5361a07e89ad47298aabe717a4f2867f: Request end of tablet copy session a24048f65ee54156b569d4bf1f738368-614d4da74aa547a28013c97f5a1079fd received from {username='slave'} at 127.22.242.69:38851
I20240503 14:14:40.976822 5794 tablet_copy_service.cc:433] P 5361a07e89ad47298aabe717a4f2867f: ending tablet copy session a24048f65ee54156b569d4bf1f738368-614d4da74aa547a28013c97f5a1079fd on tablet 614d4da74aa547a28013c97f5a1079fd with peer a24048f65ee54156b569d4bf1f738368
I20240503 14:14:40.978354 7295 raft_consensus.cc:348] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } }
I20240503 14:14:40.978832 7295 raft_consensus.cc:729] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da201109e13f4b36b3fba7477454a4c6, State: Initialized, Role: LEARNER
I20240503 14:14:40.979408 7295 consensus_queue.cc:260] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } }
I20240503 14:14:40.983194 7282 ts_tablet_manager.cc:902] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368: Initiating tablet copy from peer 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071)
I20240503 14:14:40.986869 7337 ts_tablet_manager.cc:902] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6: Initiating tablet copy from peer 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071)
I20240503 14:14:40.994784 7295 ts_tablet_manager.cc:1398] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6: Time spent starting tablet: real 0.024s user 0.008s sys 0.006s
I20240503 14:14:41.002127 7282 tablet_copy_client.cc:308] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:41071
I20240503 14:14:41.002355 5925 tablet_copy_service.cc:342] P a8a749bf14364f22a4db0f7cdfa33ce9: Request end of tablet copy session da201109e13f4b36b3fba7477454a4c6-f6b542b55eb9477ab472b3a4005e6c63 received from {username='slave'} at 127.22.242.68:54621
I20240503 14:14:41.002506 5925 tablet_copy_service.cc:433] P a8a749bf14364f22a4db0f7cdfa33ce9: ending tablet copy session da201109e13f4b36b3fba7477454a4c6-f6b542b55eb9477ab472b3a4005e6c63 on tablet f6b542b55eb9477ab472b3a4005e6c63 with peer da201109e13f4b36b3fba7477454a4c6
I20240503 14:14:41.003010 5794 tablet_copy_service.cc:140] P 5361a07e89ad47298aabe717a4f2867f: Received BeginTabletCopySession request for tablet 52983271755542f090bb26d0675ec84e from peer a24048f65ee54156b569d4bf1f738368 ({username='slave'} at 127.22.242.69:38851)
I20240503 14:14:41.003190 5794 tablet_copy_service.cc:161] P 5361a07e89ad47298aabe717a4f2867f: Beginning new tablet copy session on tablet 52983271755542f090bb26d0675ec84e from peer a24048f65ee54156b569d4bf1f738368 at {username='slave'} at 127.22.242.69:38851: session id = a24048f65ee54156b569d4bf1f738368-52983271755542f090bb26d0675ec84e
I20240503 14:14:41.005728 7270 tablet_copy_client.cc:308] T 7b8188059098481e9dd4b46858bda06c P 7bb84965b41b41fa9be23175c82e0a3a: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:36157
I20240503 14:14:41.006191 5794 tablet_copy_source_session.cc:205] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:41.008291 7282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52983271755542f090bb26d0675ec84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:41.012631 5662 tablet_copy_service.cc:140] P e5237ac15b5c4a6c88a8a07f3d44cf33: Received BeginTabletCopySession request for tablet 7b8188059098481e9dd4b46858bda06c from peer 7bb84965b41b41fa9be23175c82e0a3a ({username='slave'} at 127.22.242.70:47743)
I20240503 14:14:41.012864 5662 tablet_copy_service.cc:161] P e5237ac15b5c4a6c88a8a07f3d44cf33: Beginning new tablet copy session on tablet 7b8188059098481e9dd4b46858bda06c from peer 7bb84965b41b41fa9be23175c82e0a3a at {username='slave'} at 127.22.242.70:47743: session id = 7bb84965b41b41fa9be23175c82e0a3a-7b8188059098481e9dd4b46858bda06c
I20240503 14:14:41.016702 5662 tablet_copy_source_session.cc:205] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:41.018453 7270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b8188059098481e9dd4b46858bda06c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:41.022725 7270 tablet_copy_client.cc:783] T 7b8188059098481e9dd4b46858bda06c P 7bb84965b41b41fa9be23175c82e0a3a: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:41.022979 7270 tablet_copy_client.cc:647] T 7b8188059098481e9dd4b46858bda06c P 7bb84965b41b41fa9be23175c82e0a3a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:41.025110 7282 tablet_copy_client.cc:783] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:41.025413 7282 tablet_copy_client.cc:647] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:41.026425 7270 tablet_copy_client.cc:523] T 7b8188059098481e9dd4b46858bda06c P 7bb84965b41b41fa9be23175c82e0a3a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:41.028066 7282 tablet_copy_client.cc:523] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:41.029103 7336 ts_tablet_manager.cc:902] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368: Initiating tablet copy from peer e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157)
I20240503 14:14:41.030143 7270 tablet_bootstrap.cc:492] T 7b8188059098481e9dd4b46858bda06c P 7bb84965b41b41fa9be23175c82e0a3a: Bootstrap starting.
I20240503 14:14:41.042213 7282 tablet_bootstrap.cc:492] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368: Bootstrap starting.
I20240503 14:14:41.073432 7337 tablet_copy_client.cc:308] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:41071
I20240503 14:14:41.074468 7355 ts_tablet_manager.cc:902] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a: Initiating tablet copy from peer a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
I20240503 14:14:41.081775 5794 tablet_copy_service.cc:140] P 5361a07e89ad47298aabe717a4f2867f: Received BeginTabletCopySession request for tablet 16205fdbbb994e9f840c77dfd83dad1f from peer da201109e13f4b36b3fba7477454a4c6 ({username='slave'} at 127.22.242.68:52581)
I20240503 14:14:41.081964 5794 tablet_copy_service.cc:161] P 5361a07e89ad47298aabe717a4f2867f: Beginning new tablet copy session on tablet 16205fdbbb994e9f840c77dfd83dad1f from peer da201109e13f4b36b3fba7477454a4c6 at {username='slave'} at 127.22.242.68:52581: session id = da201109e13f4b36b3fba7477454a4c6-16205fdbbb994e9f840c77dfd83dad1f
I20240503 14:14:41.084895 5794 tablet_copy_source_session.cc:205] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:41.086810 7337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16205fdbbb994e9f840c77dfd83dad1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:41.094223 7337 tablet_copy_client.cc:783] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:41.094619 7337 tablet_copy_client.cc:647] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:41.095396 7336 tablet_copy_client.cc:308] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:36157
I20240503 14:14:41.100646 7337 tablet_copy_client.cc:523] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:41.108881 5662 tablet_copy_service.cc:140] P e5237ac15b5c4a6c88a8a07f3d44cf33: Received BeginTabletCopySession request for tablet eb339ed8f9a141caa62cc5588d07fee1 from peer a24048f65ee54156b569d4bf1f738368 ({username='slave'} at 127.22.242.69:37061)
I20240503 14:14:41.108999 7337 tablet_bootstrap.cc:492] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6: Bootstrap starting.
I20240503 14:14:41.109107 5662 tablet_copy_service.cc:161] P e5237ac15b5c4a6c88a8a07f3d44cf33: Beginning new tablet copy session on tablet eb339ed8f9a141caa62cc5588d07fee1 from peer a24048f65ee54156b569d4bf1f738368 at {username='slave'} at 127.22.242.69:37061: session id = a24048f65ee54156b569d4bf1f738368-eb339ed8f9a141caa62cc5588d07fee1
I20240503 14:14:41.112047 5662 tablet_copy_source_session.cc:205] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:41.113508 7336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb339ed8f9a141caa62cc5588d07fee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:41.126281 7336 tablet_copy_client.cc:783] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:41.126652 7336 tablet_copy_client.cc:647] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:41.126703 7355 tablet_copy_client.cc:308] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44189
I20240503 14:14:41.127549 5925 tablet_copy_service.cc:140] P a8a749bf14364f22a4db0f7cdfa33ce9: Received BeginTabletCopySession request for tablet 098804415c3b4bafa53e58119712758f from peer 7bb84965b41b41fa9be23175c82e0a3a ({username='slave'} at 127.22.242.70:37583)
I20240503 14:14:41.127714 5925 tablet_copy_service.cc:161] P a8a749bf14364f22a4db0f7cdfa33ce9: Beginning new tablet copy session on tablet 098804415c3b4bafa53e58119712758f from peer 7bb84965b41b41fa9be23175c82e0a3a at {username='slave'} at 127.22.242.70:37583: session id = 7bb84965b41b41fa9be23175c82e0a3a-098804415c3b4bafa53e58119712758f
I20240503 14:14:41.128710 7336 tablet_copy_client.cc:523] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:41.130089 7282 tablet_bootstrap.cc:492] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:41.130589 7282 tablet_bootstrap.cc:492] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368: Bootstrap complete.
I20240503 14:14:41.130573 5925 tablet_copy_source_session.cc:205] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:41.130903 7282 ts_tablet_manager.cc:1367] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368: Time spent bootstrapping tablet: real 0.089s user 0.068s sys 0.015s
I20240503 14:14:41.132004 7355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 098804415c3b4bafa53e58119712758f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:41.131966 7282 raft_consensus.cc:348] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } }
I20240503 14:14:41.132282 7282 raft_consensus.cc:729] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a24048f65ee54156b569d4bf1f738368, State: Initialized, Role: LEARNER
I20240503 14:14:41.132500 7282 consensus_queue.cc:260] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } }
I20240503 14:14:41.133132 7282 ts_tablet_manager.cc:1398] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:14:41.134004 5794 tablet_copy_service.cc:342] P 5361a07e89ad47298aabe717a4f2867f: Request end of tablet copy session a24048f65ee54156b569d4bf1f738368-52983271755542f090bb26d0675ec84e received from {username='slave'} at 127.22.242.69:38851
I20240503 14:14:41.134193 5794 tablet_copy_service.cc:433] P 5361a07e89ad47298aabe717a4f2867f: ending tablet copy session a24048f65ee54156b569d4bf1f738368-52983271755542f090bb26d0675ec84e on tablet 52983271755542f090bb26d0675ec84e with peer a24048f65ee54156b569d4bf1f738368
I20240503 14:14:41.136982 7336 tablet_bootstrap.cc:492] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368: Bootstrap starting.
I20240503 14:14:41.142467 7270 tablet_bootstrap.cc:492] T 7b8188059098481e9dd4b46858bda06c P 7bb84965b41b41fa9be23175c82e0a3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:41.142911 7270 tablet_bootstrap.cc:492] T 7b8188059098481e9dd4b46858bda06c P 7bb84965b41b41fa9be23175c82e0a3a: Bootstrap complete.
I20240503 14:14:41.143133 7355 tablet_copy_client.cc:783] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:41.143383 7355 tablet_copy_client.cc:647] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:41.143488 7270 ts_tablet_manager.cc:1367] T 7b8188059098481e9dd4b46858bda06c P 7bb84965b41b41fa9be23175c82e0a3a: Time spent bootstrapping tablet: real 0.113s user 0.054s sys 0.008s
I20240503 14:14:41.145236 7355 tablet_copy_client.cc:523] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:41.145447 7270 raft_consensus.cc:348] T 7b8188059098481e9dd4b46858bda06c P 7bb84965b41b41fa9be23175c82e0a3a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } }
I20240503 14:14:41.145745 7270 raft_consensus.cc:729] T 7b8188059098481e9dd4b46858bda06c P 7bb84965b41b41fa9be23175c82e0a3a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7bb84965b41b41fa9be23175c82e0a3a, State: Initialized, Role: LEARNER
I20240503 14:14:41.145972 7270 consensus_queue.cc:260] T 7b8188059098481e9dd4b46858bda06c P 7bb84965b41b41fa9be23175c82e0a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } }
I20240503 14:14:41.149051 7355 tablet_bootstrap.cc:492] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a: Bootstrap starting.
I20240503 14:14:41.149121 7270 ts_tablet_manager.cc:1398] T 7b8188059098481e9dd4b46858bda06c P 7bb84965b41b41fa9be23175c82e0a3a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:14:41.151916 5662 tablet_copy_service.cc:342] P e5237ac15b5c4a6c88a8a07f3d44cf33: Request end of tablet copy session 7bb84965b41b41fa9be23175c82e0a3a-7b8188059098481e9dd4b46858bda06c received from {username='slave'} at 127.22.242.70:47743
I20240503 14:14:41.152089 5662 tablet_copy_service.cc:433] P e5237ac15b5c4a6c88a8a07f3d44cf33: ending tablet copy session 7bb84965b41b41fa9be23175c82e0a3a-7b8188059098481e9dd4b46858bda06c on tablet 7b8188059098481e9dd4b46858bda06c with peer 7bb84965b41b41fa9be23175c82e0a3a
I20240503 14:14:41.180469 7355 tablet_bootstrap.cc:492] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:41.181053 7355 tablet_bootstrap.cc:492] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a: Bootstrap complete.
I20240503 14:14:41.181522 7355 ts_tablet_manager.cc:1367] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a: Time spent bootstrapping tablet: real 0.033s user 0.032s sys 0.000s
I20240503 14:14:41.182488 7355 raft_consensus.cc:348] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } }
I20240503 14:14:41.182698 7355 raft_consensus.cc:729] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7bb84965b41b41fa9be23175c82e0a3a, State: Initialized, Role: LEARNER
I20240503 14:14:41.182948 7355 consensus_queue.cc:260] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } }
I20240503 14:14:41.183441 7355 ts_tablet_manager.cc:1398] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:14:41.184756 5925 tablet_copy_service.cc:342] P a8a749bf14364f22a4db0f7cdfa33ce9: Request end of tablet copy session 7bb84965b41b41fa9be23175c82e0a3a-098804415c3b4bafa53e58119712758f received from {username='slave'} at 127.22.242.70:37583
I20240503 14:14:41.184896 5925 tablet_copy_service.cc:433] P a8a749bf14364f22a4db0f7cdfa33ce9: ending tablet copy session 7bb84965b41b41fa9be23175c82e0a3a-098804415c3b4bafa53e58119712758f on tablet 098804415c3b4bafa53e58119712758f with peer 7bb84965b41b41fa9be23175c82e0a3a
I20240503 14:14:41.188553 7337 tablet_bootstrap.cc:492] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:41.188974 7337 tablet_bootstrap.cc:492] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6: Bootstrap complete.
I20240503 14:14:41.189281 7337 ts_tablet_manager.cc:1367] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6: Time spent bootstrapping tablet: real 0.085s user 0.064s sys 0.014s
I20240503 14:14:41.190315 7337 raft_consensus.cc:348] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } }
I20240503 14:14:41.190572 7337 raft_consensus.cc:729] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da201109e13f4b36b3fba7477454a4c6, State: Initialized, Role: LEARNER
I20240503 14:14:41.190982 7337 consensus_queue.cc:260] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } }
I20240503 14:14:41.191607 7337 ts_tablet_manager.cc:1398] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:14:41.192402 5794 tablet_copy_service.cc:342] P 5361a07e89ad47298aabe717a4f2867f: Request end of tablet copy session da201109e13f4b36b3fba7477454a4c6-16205fdbbb994e9f840c77dfd83dad1f received from {username='slave'} at 127.22.242.68:52581
I20240503 14:14:41.192574 5794 tablet_copy_service.cc:433] P 5361a07e89ad47298aabe717a4f2867f: ending tablet copy session da201109e13f4b36b3fba7477454a4c6-16205fdbbb994e9f840c77dfd83dad1f on tablet 16205fdbbb994e9f840c77dfd83dad1f with peer da201109e13f4b36b3fba7477454a4c6
I20240503 14:14:41.195637 7336 tablet_bootstrap.cc:492] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:41.196074 7336 tablet_bootstrap.cc:492] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368: Bootstrap complete.
I20240503 14:14:41.196350 7336 ts_tablet_manager.cc:1367] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368: Time spent bootstrapping tablet: real 0.063s user 0.049s sys 0.009s
I20240503 14:14:41.196975 7336 raft_consensus.cc:348] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } }
I20240503 14:14:41.197176 7336 raft_consensus.cc:729] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a24048f65ee54156b569d4bf1f738368, State: Initialized, Role: LEARNER
I20240503 14:14:41.197351 7336 consensus_queue.cc:260] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } }
I20240503 14:14:41.197964 7336 ts_tablet_manager.cc:1398] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:14:41.199330 6834 raft_consensus.cc:1203] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:41.199568 5662 tablet_copy_service.cc:342] P e5237ac15b5c4a6c88a8a07f3d44cf33: Request end of tablet copy session a24048f65ee54156b569d4bf1f738368-eb339ed8f9a141caa62cc5588d07fee1 received from {username='slave'} at 127.22.242.69:37061
I20240503 14:14:41.199684 5662 tablet_copy_service.cc:433] P e5237ac15b5c4a6c88a8a07f3d44cf33: ending tablet copy session a24048f65ee54156b569d4bf1f738368-eb339ed8f9a141caa62cc5588d07fee1 on tablet eb339ed8f9a141caa62cc5588d07fee1 with peer a24048f65ee54156b569d4bf1f738368
I20240503 14:14:41.206372 6703 raft_consensus.cc:1203] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:41.358279 6964 raft_consensus.cc:1203] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:41.393827 6964 raft_consensus.cc:1203] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:41.455358 6834 raft_consensus.cc:1203] T 7b8188059098481e9dd4b46858bda06c P 7bb84965b41b41fa9be23175c82e0a3a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:41.476492 6703 raft_consensus.cc:1203] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:41.513435 6703 raft_consensus.cc:1203] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:41.549315 6834 raft_consensus.cc:1203] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:41.757138 7155 raft_consensus.cc:1050] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9: attempting to promote NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a to VOTER
I20240503 14:14:41.757938 7155 consensus_queue.cc:237] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } }
I20240503 14:14:41.760457 6834 raft_consensus.cc:1261] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a [term 1 LEARNER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:41.760481 5772 raft_consensus.cc:1261] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:41.760557 5641 raft_consensus.cc:1261] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:41.761186 6375 consensus_queue.cc:1035] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:41.761530 7155 consensus_queue.cc:1035] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:41.761840 7406 consensus_queue.cc:1035] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:41.766041 7370 raft_consensus.cc:1050] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f: attempting to promote NON_VOTER a24048f65ee54156b569d4bf1f738368 to VOTER
I20240503 14:14:41.766449 6375 raft_consensus.cc:2932] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:41.766937 7370 consensus_queue.cc:237] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } }
I20240503 14:14:41.766858 5641 raft_consensus.cc:2932] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:41.769798 5642 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:41.769181 6834 raft_consensus.cc:2932] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:41.769912 6703 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:41.770550 5904 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:41.770527 7171 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:41.771063 7171 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:41.772775 7369 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:41.776520 5495 catalog_manager.cc:5484] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33 reported cstate change: config changed from index 4 to 5, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:41.777709 5772 raft_consensus.cc:2932] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:41.785846 7171 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:41.787607 5904 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:41.789743 5642 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:41.796123 5904 consensus_queue.cc:237] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } }
I20240503 14:14:41.796262 5494 catalog_manager.cc:5484] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 reported cstate change: config changed from index 4 to 5, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:41.798686 6703 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:41.799865 6834 raft_consensus.cc:1261] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:41.800633 7460 consensus_queue.cc:1035] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:41.801364 5642 raft_consensus.cc:1261] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:41.801878 7460 consensus_queue.cc:1035] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:41.804373 7155 raft_consensus.cc:2932] T de3d8ce603a2444e8372626dd6b39b6c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:41.804978 6834 raft_consensus.cc:2932] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:41.808543 5772 consensus_queue.cc:237] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } }
I20240503 14:14:41.810185 5492 catalog_manager.cc:5484] T de3d8ce603a2444e8372626dd6b39b6c P 7bb84965b41b41fa9be23175c82e0a3a reported cstate change: config changed from index 5 to 6, VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:41.810180 5641 raft_consensus.cc:2932] T de3d8ce603a2444e8372626dd6b39b6c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:41.811157 6703 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:41.811815 7465 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:41.812605 5642 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:41.813139 7465 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:41.814313 5485 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet de3d8ce603a2444e8372626dd6b39b6c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:41.818166 7171 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:41.820264 5752 tablet_service.cc:1514] Processing DeleteTablet for tablet de3d8ce603a2444e8372626dd6b39b6c with delete_type TABLET_DATA_TOMBSTONED (TS 5361a07e89ad47298aabe717a4f2867f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46506
I20240503 14:14:41.821079 7478 tablet_replica.cc:330] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f: stopping tablet replica
I20240503 14:14:41.821556 7478 raft_consensus.cc:2229] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:41.822157 6703 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:41.823123 5642 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:41.826866 7369 raft_consensus.cc:1050] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f: attempting to promote NON_VOTER da201109e13f4b36b3fba7477454a4c6 to VOTER
I20240503 14:14:41.827178 5495 catalog_manager.cc:5484] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 reported cstate change: config changed from index 5 to 6, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:41.827720 7369 consensus_queue.cc:237] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } }
I20240503 14:14:41.830835 5884 tablet_service.cc:1514] Processing DeleteTablet for tablet 614d4da74aa547a28013c97f5a1079fd with delete_type TABLET_DATA_TOMBSTONED (TS a8a749bf14364f22a4db0f7cdfa33ce9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60268
I20240503 14:14:41.831714 5487 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 614d4da74aa547a28013c97f5a1079fd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:41.835497 6964 raft_consensus.cc:1261] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6 [term 1 LEARNER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:41.834869 5642 raft_consensus.cc:1261] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:41.836176 7171 consensus_queue.cc:1035] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:41.836349 7482 tablet_replica.cc:330] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: stopping tablet replica
I20240503 14:14:41.836450 5904 raft_consensus.cc:1261] T 16205fdbbb994e9f840c77dfd83dad1f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:41.836967 7482 raft_consensus.cc:2229] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:41.837354 7482 raft_consensus.cc:2258] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:41.837291 7465 consensus_queue.cc:1035] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:41.837808 7465 consensus_queue.cc:1035] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:41.839241 7482 ts_tablet_manager.cc:1875] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:41.845139 7478 raft_consensus.cc:2258] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:41.847465 7478 ts_tablet_manager.cc:1875] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:41.847483 7482 ts_tablet_manager.cc:1888] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:41.847645 7482 log.cc:1196] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/614d4da74aa547a28013c97f5a1079fd
I20240503 14:14:41.849967 7465 raft_consensus.cc:2932] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:41.850805 5905 raft_consensus.cc:2932] T 16205fdbbb994e9f840c77dfd83dad1f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:41.853935 5641 raft_consensus.cc:2932] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:41.854400 5492 catalog_manager.cc:5484] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f reported cstate change: config changed from index 4 to 5, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:41.855526 5485 catalog_manager.cc:4830] TS a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189): tablet 614d4da74aa547a28013c97f5a1079fd (table rebalance_test_table_1 [id=a268a544233045a1b8bcb56727a9bbb8]) successfully deleted
I20240503 14:14:41.864486 7478 ts_tablet_manager.cc:1888] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:41.864645 7478 log.cc:1196] T de3d8ce603a2444e8372626dd6b39b6c P 5361a07e89ad47298aabe717a4f2867f: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/de3d8ce603a2444e8372626dd6b39b6c
I20240503 14:14:41.865623 5487 catalog_manager.cc:4830] TS 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071): tablet de3d8ce603a2444e8372626dd6b39b6c (table rebalance_test_table_2 [id=632b93f2c4174b0080e62682cf1200b4]) successfully deleted
I20240503 14:14:41.865240 6964 raft_consensus.cc:2932] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:41.868273 5772 consensus_queue.cc:237] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } }
I20240503 14:14:41.870570 5904 raft_consensus.cc:1261] T 16205fdbbb994e9f840c77dfd83dad1f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:41.870622 6964 raft_consensus.cc:1261] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:41.871887 7369 consensus_queue.cc:1035] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:41.872172 7370 consensus_queue.cc:1035] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:41.874930 7369 raft_consensus.cc:2932] T 16205fdbbb994e9f840c77dfd83dad1f P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:41.875524 5904 raft_consensus.cc:2932] T 16205fdbbb994e9f840c77dfd83dad1f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:41.875586 6964 raft_consensus.cc:2932] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:41.879770 5487 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 16205fdbbb994e9f840c77dfd83dad1f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:41.880436 5495 catalog_manager.cc:5484] T 16205fdbbb994e9f840c77dfd83dad1f P da201109e13f4b36b3fba7477454a4c6 reported cstate change: config changed from index 5 to 6, VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:41.885564 5620 tablet_service.cc:1514] Processing DeleteTablet for tablet 16205fdbbb994e9f840c77dfd83dad1f with delete_type TABLET_DATA_TOMBSTONED (TS e5237ac15b5c4a6c88a8a07f3d44cf33 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33472
I20240503 14:14:41.886433 7494 tablet_replica.cc:330] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33: stopping tablet replica
I20240503 14:14:41.886904 7494 raft_consensus.cc:2229] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:41.887339 7494 raft_consensus.cc:2258] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:41.889097 7494 ts_tablet_manager.cc:1875] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:41.893857 7171 raft_consensus.cc:1050] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f: attempting to promote NON_VOTER a24048f65ee54156b569d4bf1f738368 to VOTER
I20240503 14:14:41.894127 7155 raft_consensus.cc:1050] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9: attempting to promote NON_VOTER da201109e13f4b36b3fba7477454a4c6 to VOTER
I20240503 14:14:41.894784 7171 consensus_queue.cc:237] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } }
I20240503 14:14:41.895016 7155 consensus_queue.cc:237] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } }
I20240503 14:14:41.897300 6964 raft_consensus.cc:1261] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6 [term 1 LEARNER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:41.897647 5641 raft_consensus.cc:1261] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:41.897725 5772 raft_consensus.cc:1261] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:41.898317 6375 consensus_queue.cc:1035] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:41.898594 7406 consensus_queue.cc:1035] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:41.899986 7406 consensus_queue.cc:1035] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:41.902716 7494 ts_tablet_manager.cc:1888] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:41.907485 6703 raft_consensus.cc:1261] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:41.908334 7465 consensus_queue.cc:1035] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:41.916123 5905 raft_consensus.cc:1261] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:41.917068 7465 consensus_queue.cc:1035] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:41.924284 5642 raft_consensus.cc:1261] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:41.924353 7494 log.cc:1196] T 16205fdbbb994e9f840c77dfd83dad1f P e5237ac15b5c4a6c88a8a07f3d44cf33: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/16205fdbbb994e9f840c77dfd83dad1f
I20240503 14:14:41.924867 7171 consensus_queue.cc:1035] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:41.927134 7465 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:41.933280 6703 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:41.937171 5497 catalog_manager.cc:5484] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f reported cstate change: config changed from index 4 to 5, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:41.938356 5905 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:41.944458 7195 raft_consensus.cc:1050] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33: attempting to promote NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a to VOTER
I20240503 14:14:41.945525 7195 consensus_queue.cc:237] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } }
I20240503 14:14:41.950448 5485 catalog_manager.cc:4830] TS e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157): tablet 16205fdbbb994e9f840c77dfd83dad1f (table rebalance_test_table_2 [id=632b93f2c4174b0080e62682cf1200b4]) successfully deleted
I20240503 14:14:41.952054 7406 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:41.955731 6834 raft_consensus.cc:1261] T 7b8188059098481e9dd4b46858bda06c P 7bb84965b41b41fa9be23175c82e0a3a [term 1 LEARNER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:41.956341 7408 consensus_queue.cc:1035] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:41.956647 5641 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:41.959182 5905 raft_consensus.cc:1261] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:41.960198 7407 consensus_queue.cc:1035] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:41.960517 5772 raft_consensus.cc:1261] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:41.961154 7407 consensus_queue.cc:1035] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:41.963471 5773 consensus_queue.cc:237] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } }
I20240503 14:14:41.965534 5642 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:41.966673 5773 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:41.970155 5492 catalog_manager.cc:5484] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 reported cstate change: config changed from index 4 to 5, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:41.971230 5641 raft_consensus.cc:1261] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:41.971740 6703 raft_consensus.cc:1261] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:41.971933 7369 consensus_queue.cc:1035] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:41.972739 7369 consensus_queue.cc:1035] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:41.979841 7407 raft_consensus.cc:2932] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:41.980799 6964 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:41.991987 5774 raft_consensus.cc:2932] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:41.996865 5904 consensus_queue.cc:237] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } }
I20240503 14:14:41.997032 7465 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:41.998677 6964 raft_consensus.cc:1261] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:41.999320 6375 consensus_queue.cc:1035] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:42.000337 5773 raft_consensus.cc:1261] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:42.001174 6375 consensus_queue.cc:1035] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:42.001253 5487 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 52983271755542f090bb26d0675ec84e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:42.001854 6375 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:42.002493 6964 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:42.002827 6702 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:42.008180 5497 catalog_manager.cc:5484] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368 reported cstate change: config changed from index 5 to 6, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:42.008183 5492 catalog_manager.cc:5484] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6 reported cstate change: config changed from index 5 to 6, VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:42.012286 5620 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b542b55eb9477ab472b3a4005e6c63 with delete_type TABLET_DATA_TOMBSTONED (TS e5237ac15b5c4a6c88a8a07f3d44cf33 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33472
I20240503 14:14:42.012727 7494 tablet_replica.cc:330] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33: stopping tablet replica
I20240503 14:14:42.012935 7494 raft_consensus.cc:2229] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:42.013258 7494 raft_consensus.cc:2258] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:42.014336 7494 ts_tablet_manager.cc:1875] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:42.015058 5774 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:42.015749 5884 tablet_service.cc:1514] Processing DeleteTablet for tablet 52983271755542f090bb26d0675ec84e with delete_type TABLET_DATA_TOMBSTONED (TS a8a749bf14364f22a4db0f7cdfa33ce9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60268
I20240503 14:14:42.016129 7482 tablet_replica.cc:330] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9: stopping tablet replica
I20240503 14:14:42.016362 7482 raft_consensus.cc:2229] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:42.016503 5485 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f6b542b55eb9477ab472b3a4005e6c63 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:42.016755 7482 raft_consensus.cc:2258] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:42.018039 7482 ts_tablet_manager.cc:1875] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:42.017644 6834 raft_consensus.cc:2932] T 7b8188059098481e9dd4b46858bda06c P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:42.024240 7482 ts_tablet_manager.cc:1888] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:42.024456 7482 log.cc:1196] T 52983271755542f090bb26d0675ec84e P a8a749bf14364f22a4db0f7cdfa33ce9: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/52983271755542f090bb26d0675ec84e
I20240503 14:14:42.025363 5485 catalog_manager.cc:4830] TS a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189): tablet 52983271755542f090bb26d0675ec84e (table rebalance_test_table_1 [id=a268a544233045a1b8bcb56727a9bbb8]) successfully deleted
I20240503 14:14:42.026572 5497 catalog_manager.cc:5484] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f reported cstate change: config changed from index 4 to 5, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:42.027240 7494 ts_tablet_manager.cc:1888] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:42.027377 7494 log.cc:1196] T f6b542b55eb9477ab472b3a4005e6c63 P e5237ac15b5c4a6c88a8a07f3d44cf33: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/f6b542b55eb9477ab472b3a4005e6c63
I20240503 14:14:42.027549 5641 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:42.029659 5485 catalog_manager.cc:4830] TS e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157): tablet f6b542b55eb9477ab472b3a4005e6c63 (table rebalance_test_table_0 [id=49c4626b38eb4b7194cd2add19c0cb32]) successfully deleted
I20240503 14:14:42.039913 5904 raft_consensus.cc:2932] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:42.047215 5641 consensus_queue.cc:237] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } }
I20240503 14:14:42.048940 6834 raft_consensus.cc:1261] T 7b8188059098481e9dd4b46858bda06c P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:42.049067 5904 raft_consensus.cc:1261] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:42.049641 7407 consensus_queue.cc:1035] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:42.049965 7195 consensus_queue.cc:1035] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:42.052372 7517 raft_consensus.cc:2932] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:42.053161 6834 raft_consensus.cc:2932] T 7b8188059098481e9dd4b46858bda06c P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:42.056380 5485 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7b8188059098481e9dd4b46858bda06c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:42.059762 5904 raft_consensus.cc:2932] T 7b8188059098481e9dd4b46858bda06c P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:42.059607 5497 catalog_manager.cc:5484] T 7b8188059098481e9dd4b46858bda06c P e5237ac15b5c4a6c88a8a07f3d44cf33 reported cstate change: config changed from index 5 to 6, VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:42.060950 7460 raft_consensus.cc:1050] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9: attempting to promote NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a to VOTER
I20240503 14:14:42.061789 7460 consensus_queue.cc:237] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } }
I20240503 14:14:42.065191 5752 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b8188059098481e9dd4b46858bda06c with delete_type TABLET_DATA_TOMBSTONED (TS 5361a07e89ad47298aabe717a4f2867f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46506
I20240503 14:14:42.065524 7478 tablet_replica.cc:330] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f: stopping tablet replica
I20240503 14:14:42.065713 7478 raft_consensus.cc:2229] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:42.066118 7478 raft_consensus.cc:2258] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:42.067521 7478 ts_tablet_manager.cc:1875] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:42.075453 7478 ts_tablet_manager.cc:1888] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:42.075585 7478 log.cc:1196] T 7b8188059098481e9dd4b46858bda06c P 5361a07e89ad47298aabe717a4f2867f: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/7b8188059098481e9dd4b46858bda06c
I20240503 14:14:42.076352 5487 catalog_manager.cc:4830] TS 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071): tablet 7b8188059098481e9dd4b46858bda06c (table rebalance_test_table_0 [id=49c4626b38eb4b7194cd2add19c0cb32]) successfully deleted
I20240503 14:14:42.076766 5774 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:42.077025 6834 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a [term 1 LEARNER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:42.077028 5641 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:42.077410 7460 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:42.077772 7406 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:42.078130 7155 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:42.082259 7517 raft_consensus.cc:1050] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33: attempting to promote NON_VOTER a24048f65ee54156b569d4bf1f738368 to VOTER
I20240503 14:14:42.083359 7517 consensus_queue.cc:237] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } }
I20240503 14:14:42.106563 6702 raft_consensus.cc:1261] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:42.107368 7195 consensus_queue.cc:1035] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:42.115278 5904 raft_consensus.cc:1261] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:42.116214 7408 consensus_queue.cc:1035] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:42.117725 7460 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:42.118850 5641 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:42.125057 5774 raft_consensus.cc:1261] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:42.125666 7195 consensus_queue.cc:1035] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:42.125274 6834 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:42.125715 5492 catalog_manager.cc:5484] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 reported cstate change: config changed from index 4 to 5, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:42.126991 5772 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:42.132558 7195 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:42.135888 5905 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:42.139719 6702 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:42.140384 5903 consensus_queue.cc:237] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } }
I20240503 14:14:42.142817 6834 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:42.143254 5495 catalog_manager.cc:5484] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 reported cstate change: config changed from index 4 to 5, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:42.143829 5641 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:42.144531 7155 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:42.144868 7460 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:42.147521 6375 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:42.148069 5641 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:42.148397 6834 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:42.151916 5485 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 098804415c3b4bafa53e58119712758f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:42.153163 5642 consensus_queue.cc:237] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } }
I20240503 14:14:42.153836 5492 catalog_manager.cc:5484] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a reported cstate change: config changed from index 5 to 6, VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:42.154126 5774 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:42.155045 6702 raft_consensus.cc:1261] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:42.155668 7517 consensus_queue.cc:1035] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:42.157167 5752 tablet_service.cc:1514] Processing DeleteTablet for tablet 098804415c3b4bafa53e58119712758f with delete_type TABLET_DATA_TOMBSTONED (TS 5361a07e89ad47298aabe717a4f2867f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46506
I20240503 14:14:42.157526 7478 tablet_replica.cc:330] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: stopping tablet replica
I20240503 14:14:42.157727 7478 raft_consensus.cc:2229] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:42.157936 5772 raft_consensus.cc:1261] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:42.158030 7478 raft_consensus.cc:2258] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:42.159236 7478 ts_tablet_manager.cc:1875] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:42.159370 7195 consensus_queue.cc:1035] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:42.161713 7517 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:42.165906 7478 ts_tablet_manager.cc:1888] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:42.166051 7478 log.cc:1196] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/098804415c3b4bafa53e58119712758f
I20240503 14:14:42.165948 5772 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:42.166796 5487 catalog_manager.cc:4830] TS 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071): tablet 098804415c3b4bafa53e58119712758f (table rebalance_test_table_2 [id=632b93f2c4174b0080e62682cf1200b4]) successfully deleted
I20240503 14:14:42.168058 5485 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb339ed8f9a141caa62cc5588d07fee1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:42.168762 5494 catalog_manager.cc:5484] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 reported cstate change: config changed from index 5 to 6, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:42.169088 6702 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:42.173369 5884 tablet_service.cc:1514] Processing DeleteTablet for tablet eb339ed8f9a141caa62cc5588d07fee1 with delete_type TABLET_DATA_TOMBSTONED (TS a8a749bf14364f22a4db0f7cdfa33ce9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60268
I20240503 14:14:42.173718 7482 tablet_replica.cc:330] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: stopping tablet replica
I20240503 14:14:42.173882 7482 raft_consensus.cc:2229] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:42.174300 7482 raft_consensus.cc:2258] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:42.175875 7482 ts_tablet_manager.cc:1875] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:42.181519 7482 ts_tablet_manager.cc:1888] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:42.181618 7482 log.cc:1196] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/eb339ed8f9a141caa62cc5588d07fee1
I20240503 14:14:42.182240 5485 catalog_manager.cc:4830] TS a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189): tablet eb339ed8f9a141caa62cc5588d07fee1 (table rebalance_test_table_1 [id=a268a544233045a1b8bcb56727a9bbb8]) successfully deleted
I20240503 14:14:42.283780 5752 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:42.301369 6814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:42.304236 5884 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:42.307816 5620 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:42.317003 6944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:42.325063 6683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:42.404430 5641 consensus_queue.cc:237] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } }
I20240503 14:14:42.406819 5772 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:42.407783 5904 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:42.408083 7195 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:42.408468 7408 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:42.411340 7195 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } } }
I20240503 14:14:42.413846 5904 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } } }
W20240503 14:14:42.417603 5575 consensus_peers.cc:485] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 -> Peer da201109e13f4b36b3fba7477454a4c6 (127.22.242.68:42505): Couldn't send request to peer da201109e13f4b36b3fba7477454a4c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bfd4f9dbf8a946c6bb2c2b37210efe55. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:42.419407 5494 catalog_manager.cc:5484] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 reported cstate change: config changed from index -1 to 4, NON_VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) added. New cstate: current_term: 1 leader_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:42.424865 5772 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } } }
I20240503 14:14:42.874677 7617 ts_tablet_manager.cc:902] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6: Initiating tablet copy from peer e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157)
I20240503 14:14:42.876303 7617 tablet_copy_client.cc:308] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:36157
I20240503 14:14:42.881742 5662 tablet_copy_service.cc:140] P e5237ac15b5c4a6c88a8a07f3d44cf33: Received BeginTabletCopySession request for tablet bfd4f9dbf8a946c6bb2c2b37210efe55 from peer da201109e13f4b36b3fba7477454a4c6 ({username='slave'} at 127.22.242.68:36125)
I20240503 14:14:42.881889 5662 tablet_copy_service.cc:161] P e5237ac15b5c4a6c88a8a07f3d44cf33: Beginning new tablet copy session on tablet bfd4f9dbf8a946c6bb2c2b37210efe55 from peer da201109e13f4b36b3fba7477454a4c6 at {username='slave'} at 127.22.242.68:36125: session id = da201109e13f4b36b3fba7477454a4c6-bfd4f9dbf8a946c6bb2c2b37210efe55
I20240503 14:14:42.883841 5662 tablet_copy_source_session.cc:205] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:42.885040 7617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd4f9dbf8a946c6bb2c2b37210efe55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:42.889573 7617 tablet_copy_client.cc:783] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:42.889812 7617 tablet_copy_client.cc:647] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:42.891405 7617 tablet_copy_client.cc:523] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:42.893646 7617 tablet_bootstrap.cc:492] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6: Bootstrap starting.
I20240503 14:14:42.922202 7617 tablet_bootstrap.cc:492] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:42.922627 7617 tablet_bootstrap.cc:492] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6: Bootstrap complete.
I20240503 14:14:42.922927 7617 ts_tablet_manager.cc:1367] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6: Time spent bootstrapping tablet: real 0.029s user 0.027s sys 0.001s
I20240503 14:14:42.923570 7617 raft_consensus.cc:348] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } }
I20240503 14:14:42.923771 7617 raft_consensus.cc:729] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da201109e13f4b36b3fba7477454a4c6, State: Initialized, Role: LEARNER
I20240503 14:14:42.923980 7617 consensus_queue.cc:260] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } }
I20240503 14:14:42.924793 7617 ts_tablet_manager.cc:1398] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:14:42.925557 5662 tablet_copy_service.cc:342] P e5237ac15b5c4a6c88a8a07f3d44cf33: Request end of tablet copy session da201109e13f4b36b3fba7477454a4c6-bfd4f9dbf8a946c6bb2c2b37210efe55 received from {username='slave'} at 127.22.242.68:36125
I20240503 14:14:42.925698 5662 tablet_copy_service.cc:433] P e5237ac15b5c4a6c88a8a07f3d44cf33: ending tablet copy session da201109e13f4b36b3fba7477454a4c6-bfd4f9dbf8a946c6bb2c2b37210efe55 on tablet bfd4f9dbf8a946c6bb2c2b37210efe55 with peer da201109e13f4b36b3fba7477454a4c6
I20240503 14:14:43.308034 6964 raft_consensus.cc:1203] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:43.774631 7517 raft_consensus.cc:1050] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: attempting to promote NON_VOTER da201109e13f4b36b3fba7477454a4c6 to VOTER
I20240503 14:14:43.775275 7517 consensus_queue.cc:237] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } }
I20240503 14:14:43.777397 6964 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6 [term 1 LEARNER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.777479 5772 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.777487 5903 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.778036 7648 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.778373 7517 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.778654 7195 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.782547 7517 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:43.784682 5903 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:43.788040 5772 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:43.788806 5494 catalog_manager.cc:5484] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 reported cstate change: config changed from index 4 to 5, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:43.790315 6963 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:43.919896 5641 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bfd4f9dbf8a946c6bb2c2b37210efe55"
dest_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33050
I20240503 14:14:43.920137 5641 raft_consensus.cc:595] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:44.137161 5641 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bfd4f9dbf8a946c6bb2c2b37210efe55"
dest_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33064
I20240503 14:14:44.137364 5641 raft_consensus.cc:595] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:44.279165 7517 raft_consensus.cc:979] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: : Instructing follower 5361a07e89ad47298aabe717a4f2867f to start an election
I20240503 14:14:44.279364 7517 raft_consensus.cc:1067] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Signalling peer 5361a07e89ad47298aabe717a4f2867f to start an election
I20240503 14:14:44.280205 5772 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bfd4f9dbf8a946c6bb2c2b37210efe55"
dest_uuid: "5361a07e89ad47298aabe717a4f2867f"
from {username='slave'} at 127.22.242.65:54431
I20240503 14:14:44.280431 5772 raft_consensus.cc:482] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:44.280534 5772 raft_consensus.cc:3037] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:44.282373 5772 raft_consensus.cc:504] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } }
I20240503 14:14:44.283159 5772 leader_election.cc:288] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [CANDIDATE]: Term 2 election: Requested vote from peers e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157), a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189), da201109e13f4b36b3fba7477454a4c6 (127.22.242.68:42505)
I20240503 14:14:44.283519 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd4f9dbf8a946c6bb2c2b37210efe55" candidate_uuid: "5361a07e89ad47298aabe717a4f2867f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33"
I20240503 14:14:44.283625 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd4f9dbf8a946c6bb2c2b37210efe55" candidate_uuid: "5361a07e89ad47298aabe717a4f2867f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9"
I20240503 14:14:44.283754 5641 raft_consensus.cc:3032] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:44.283849 5641 raft_consensus.cc:729] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e5237ac15b5c4a6c88a8a07f3d44cf33, State: Running, Role: LEADER
I20240503 14:14:44.283874 5903 raft_consensus.cc:3037] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:44.283797 6963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd4f9dbf8a946c6bb2c2b37210efe55" candidate_uuid: "5361a07e89ad47298aabe717a4f2867f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "da201109e13f4b36b3fba7477454a4c6"
I20240503 14:14:44.284128 6963 raft_consensus.cc:3037] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:44.284152 5641 consensus_queue.cc:260] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } }
I20240503 14:14:44.284684 5641 raft_consensus.cc:3037] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:44.286473 5641 raft_consensus.cc:2454] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5361a07e89ad47298aabe717a4f2867f in term 2.
I20240503 14:14:44.286636 5903 raft_consensus.cc:2454] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5361a07e89ad47298aabe717a4f2867f in term 2.
I20240503 14:14:44.286944 6963 raft_consensus.cc:2454] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5361a07e89ad47298aabe717a4f2867f in term 2.
I20240503 14:14:44.287048 5707 leader_election.cc:302] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5361a07e89ad47298aabe717a4f2867f, a8a749bf14364f22a4db0f7cdfa33ce9, e5237ac15b5c4a6c88a8a07f3d44cf33; no voters:
I20240503 14:14:44.287354 7641 raft_consensus.cc:2781] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:44.287515 7641 raft_consensus.cc:686] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 2 LEADER]: Becoming Leader. State: Replica: 5361a07e89ad47298aabe717a4f2867f, State: Running, Role: LEADER
I20240503 14:14:44.287811 7641 consensus_queue.cc:237] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } }
I20240503 14:14:44.290176 5497 catalog_manager.cc:5484] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f reported cstate change: term changed from 1 to 2, leader changed from e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) to 5361a07e89ad47298aabe717a4f2867f (127.22.242.66). New cstate: current_term: 2 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:44.750540 6963 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6 [term 2 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:14:44.751077 7641 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:44.844380 5903 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 2 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:14:44.845069 7641 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:44.849833 5641 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 2 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:14:44.851447 7697 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:44.854430 5772 consensus_queue.cc:237] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } }
I20240503 14:14:44.856117 6963 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6 [term 2 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:14:44.856182 5903 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 2 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:14:44.856633 7641 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:44.856877 7697 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:44.859086 7641 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:44.859688 5903 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:44.860396 6963 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:44.862877 5487 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bfd4f9dbf8a946c6bb2c2b37210efe55 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:44.864471 5495 catalog_manager.cc:5484] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f reported cstate change: config changed from index 5 to 7, VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) evicted. New cstate: current_term: 2 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:44.868765 5620 tablet_service.cc:1514] Processing DeleteTablet for tablet bfd4f9dbf8a946c6bb2c2b37210efe55 with delete_type TABLET_DATA_TOMBSTONED (TS e5237ac15b5c4a6c88a8a07f3d44cf33 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33472
I20240503 14:14:44.869381 7704 tablet_replica.cc:330] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: stopping tablet replica
I20240503 14:14:44.869580 7704 raft_consensus.cc:2229] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:44.869774 7704 raft_consensus.cc:2258] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:44.870710 7704 ts_tablet_manager.cc:1875] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:44.874934 7704 ts_tablet_manager.cc:1888] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:14:44.875053 7704 log.cc:1196] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/bfd4f9dbf8a946c6bb2c2b37210efe55
I20240503 14:14:44.875703 5485 catalog_manager.cc:4830] TS e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157): tablet bfd4f9dbf8a946c6bb2c2b37210efe55 (table rebalance_test_table_0 [id=49c4626b38eb4b7194cd2add19c0cb32]) successfully deleted
I20240503 14:14:45.064558 5752 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:14:45.075647 6814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:45.076736 6683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:45.103600 5620 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:45.118022 6944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:45.119485 5884 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:45.286948 5752 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:14:45.296612 6683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:45.308687 6814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:45.316324 6944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:45.321856 5620 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:45.326269 5884 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:45.405885 5772 consensus_queue.cc:237] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } }
I20240503 14:14:45.408870 5641 raft_consensus.cc:1261] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.409621 6702 raft_consensus.cc:1261] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.409600 7694 consensus_queue.cc:1035] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.410248 5772 consensus_queue.cc:237] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } }
I20240503 14:14:45.410290 7694 consensus_queue.cc:1035] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:14:45.419389 5708 consensus_peers.cc:485] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f -> Peer 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70:43863): Couldn't send request to peer 7bb84965b41b41fa9be23175c82e0a3a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52983271755542f090bb26d0675ec84e. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:45.421517 5707 consensus_peers.cc:485] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f -> Peer a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189): Couldn't send request to peer a8a749bf14364f22a4db0f7cdfa33ce9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:45.421607 5642 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.422019 6703 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.422272 7796 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.422333 7641 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } } }
I20240503 14:14:45.422928 7694 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.424427 5642 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } } }
I20240503 14:14:45.432240 5497 catalog_manager.cc:5484] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 reported cstate change: config changed from index 6 to 7, NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) added. New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } } }
I20240503 14:14:45.440497 6703 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } } }
I20240503 14:14:45.452603 7641 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } }
I20240503 14:14:45.453569 5641 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } }
I20240503 14:14:45.459591 5492 catalog_manager.cc:5484] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 reported cstate change: config changed from index 6 to 7, NON_VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) added. New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } }
I20240503 14:14:45.464107 6702 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } }
I20240503 14:14:45.477641 5903 consensus_queue.cc:237] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } }
I20240503 14:14:45.490569 6963 raft_consensus.cc:1261] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.492414 6375 consensus_queue.cc:1035] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.495947 5772 raft_consensus.cc:1261] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.496840 6375 consensus_queue.cc:1035] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.504483 6375 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } }
I20240503 14:14:45.507818 5772 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } }
I20240503 14:14:45.509546 6964 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } }
I20240503 14:14:45.514379 5495 catalog_manager.cc:5484] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 reported cstate change: config changed from index 6 to 7, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:14:45.517055 5841 consensus_peers.cc:485] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 -> Peer a24048f65ee54156b569d4bf1f738368 (127.22.242.69:42145): Couldn't send request to peer a24048f65ee54156b569d4bf1f738368. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6b542b55eb9477ab472b3a4005e6c63. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:45.521660 5641 consensus_queue.cc:237] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } }
W20240503 14:14:45.525736 5575 consensus_peers.cc:485] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 -> Peer a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189): Couldn't send request to peer a8a749bf14364f22a4db0f7cdfa33ce9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:45.526921 6702 raft_consensus.cc:1261] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.527765 5774 raft_consensus.cc:1261] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.527907 7517 consensus_queue.cc:1035] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.529289 7517 consensus_queue.cc:1035] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.533849 7517 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } }
I20240503 14:14:45.541183 5772 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } }
I20240503 14:14:45.542129 6702 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } }
I20240503 14:14:45.548317 5495 catalog_manager.cc:5484] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f reported cstate change: config changed from index 6 to 7, NON_VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) added. New cstate: current_term: 1 leader_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } }
I20240503 14:14:45.560329 5903 consensus_queue.cc:237] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } }
I20240503 14:14:45.563254 6834 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.563266 5641 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 14:14:45.563804 5841 consensus_peers.cc:485] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 -> Peer a24048f65ee54156b569d4bf1f738368 (127.22.242.69:42145): Couldn't send request to peer a24048f65ee54156b569d4bf1f738368. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 098804415c3b4bafa53e58119712758f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:45.563895 7822 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.564184 7817 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.564646 5772 consensus_queue.cc:237] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: NON_VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: true } }
W20240503 14:14:45.567759 5707 consensus_peers.cc:485] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f -> Peer e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157): Couldn't send request to peer e5237ac15b5c4a6c88a8a07f3d44cf33. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:45.568881 6964 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6 [term 2 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:14:45.568979 5903 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 2 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:14:45.569559 7694 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:45.570096 7694 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:45.572616 7641 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: NON_VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: true } } }
I20240503 14:14:45.573532 5904 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: NON_VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: true } } }
I20240503 14:14:45.576738 6964 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: NON_VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: true } } }
I20240503 14:14:45.579597 5492 catalog_manager.cc:5484] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 reported cstate change: config changed from index 7 to 8, NON_VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) added. New cstate: current_term: 2 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: NON_VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: true } } }
I20240503 14:14:45.592845 7817 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } }
I20240503 14:14:45.593868 5641 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } }
I20240503 14:14:45.599301 6834 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } }
I20240503 14:14:45.599712 5497 catalog_manager.cc:5484] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 reported cstate change: config changed from index 6 to 7, NON_VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } }
I20240503 14:14:45.623544 5772 consensus_queue.cc:237] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } }
I20240503 14:14:45.626056 6702 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:45.625906 5903 consensus_queue.cc:237] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: NON_VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: true } }
I20240503 14:14:45.626571 7694 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:45.628580 6834 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:45.629431 5641 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:45.629074 7641 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } } }
I20240503 14:14:45.629654 6702 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } } }
I20240503 14:14:45.630158 7694 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:45.630263 7817 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:45.631326 5642 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:45.631650 5641 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } } }
W20240503 14:14:45.632486 5841 consensus_peers.cc:485] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 -> Peer a24048f65ee54156b569d4bf1f738368 (127.22.242.69:42145): Couldn't send request to peer a24048f65ee54156b569d4bf1f738368. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 098804415c3b4bafa53e58119712758f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:45.632673 7817 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:14:45.634820 5707 consensus_peers.cc:485] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f -> Peer a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189): Couldn't send request to peer a8a749bf14364f22a4db0f7cdfa33ce9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:45.635040 5707 consensus_peers.cc:485] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f -> Peer da201109e13f4b36b3fba7477454a4c6 (127.22.242.68:42505): Couldn't send request to peer da201109e13f4b36b3fba7477454a4c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 614d4da74aa547a28013c97f5a1079fd. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:45.635058 5492 catalog_manager.cc:5484] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 reported cstate change: config changed from index 7 to 8, NON_VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) added. New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } } }
W20240503 14:14:45.636411 5840 consensus_peers.cc:485] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 -> Peer 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071): Couldn't send request to peer 5361a07e89ad47298aabe717a4f2867f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:45.637542 7817 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: NON_VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: true } } }
I20240503 14:14:45.640633 5641 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: NON_VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: true } } }
I20240503 14:14:45.641891 6834 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: NON_VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: true } } }
I20240503 14:14:45.647187 5495 catalog_manager.cc:5484] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 reported cstate change: config changed from index 7 to 8, NON_VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) added. New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: NON_VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: true } } }
I20240503 14:14:45.816324 7895 ts_tablet_manager.cc:902] T 52983271755542f090bb26d0675ec84e P 7bb84965b41b41fa9be23175c82e0a3a: Initiating tablet copy from peer 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071)
I20240503 14:14:45.817727 7895 tablet_copy_client.cc:308] T 52983271755542f090bb26d0675ec84e P 7bb84965b41b41fa9be23175c82e0a3a: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:41071
I20240503 14:14:45.823922 5794 tablet_copy_service.cc:140] P 5361a07e89ad47298aabe717a4f2867f: Received BeginTabletCopySession request for tablet 52983271755542f090bb26d0675ec84e from peer 7bb84965b41b41fa9be23175c82e0a3a ({username='slave'} at 127.22.242.70:40489)
I20240503 14:14:45.824136 5794 tablet_copy_service.cc:161] P 5361a07e89ad47298aabe717a4f2867f: Beginning new tablet copy session on tablet 52983271755542f090bb26d0675ec84e from peer 7bb84965b41b41fa9be23175c82e0a3a at {username='slave'} at 127.22.242.70:40489: session id = 7bb84965b41b41fa9be23175c82e0a3a-52983271755542f090bb26d0675ec84e
I20240503 14:14:45.826117 5794 tablet_copy_source_session.cc:205] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:45.827363 7895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52983271755542f090bb26d0675ec84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:45.831983 7895 tablet_copy_client.cc:783] T 52983271755542f090bb26d0675ec84e P 7bb84965b41b41fa9be23175c82e0a3a: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:45.832259 7895 tablet_copy_client.cc:647] T 52983271755542f090bb26d0675ec84e P 7bb84965b41b41fa9be23175c82e0a3a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:45.833930 7895 tablet_copy_client.cc:523] T 52983271755542f090bb26d0675ec84e P 7bb84965b41b41fa9be23175c82e0a3a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:45.836431 7895 tablet_bootstrap.cc:492] T 52983271755542f090bb26d0675ec84e P 7bb84965b41b41fa9be23175c82e0a3a: Bootstrap starting.
I20240503 14:14:45.866212 7895 tablet_bootstrap.cc:492] T 52983271755542f090bb26d0675ec84e P 7bb84965b41b41fa9be23175c82e0a3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:45.866616 7895 tablet_bootstrap.cc:492] T 52983271755542f090bb26d0675ec84e P 7bb84965b41b41fa9be23175c82e0a3a: Bootstrap complete.
I20240503 14:14:45.866845 7895 ts_tablet_manager.cc:1367] T 52983271755542f090bb26d0675ec84e P 7bb84965b41b41fa9be23175c82e0a3a: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.004s
I20240503 14:14:45.867619 7895 raft_consensus.cc:348] T 52983271755542f090bb26d0675ec84e P 7bb84965b41b41fa9be23175c82e0a3a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } }
I20240503 14:14:45.867830 7895 raft_consensus.cc:729] T 52983271755542f090bb26d0675ec84e P 7bb84965b41b41fa9be23175c82e0a3a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7bb84965b41b41fa9be23175c82e0a3a, State: Initialized, Role: LEARNER
I20240503 14:14:45.868002 7895 consensus_queue.cc:260] T 52983271755542f090bb26d0675ec84e P 7bb84965b41b41fa9be23175c82e0a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: NON_VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: true } }
I20240503 14:14:45.868422 7895 ts_tablet_manager.cc:1398] T 52983271755542f090bb26d0675ec84e P 7bb84965b41b41fa9be23175c82e0a3a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 14:14:45.869356 5794 tablet_copy_service.cc:342] P 5361a07e89ad47298aabe717a4f2867f: Request end of tablet copy session 7bb84965b41b41fa9be23175c82e0a3a-52983271755542f090bb26d0675ec84e received from {username='slave'} at 127.22.242.70:40489
I20240503 14:14:45.869467 5794 tablet_copy_service.cc:433] P 5361a07e89ad47298aabe717a4f2867f: ending tablet copy session 7bb84965b41b41fa9be23175c82e0a3a-52983271755542f090bb26d0675ec84e on tablet 52983271755542f090bb26d0675ec84e with peer 7bb84965b41b41fa9be23175c82e0a3a
I20240503 14:14:45.941388 7927 ts_tablet_manager.cc:902] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: Initiating tablet copy from peer e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157)
I20240503 14:14:45.970157 7927 tablet_copy_client.cc:308] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:36157
I20240503 14:14:45.971398 5662 tablet_copy_service.cc:140] P e5237ac15b5c4a6c88a8a07f3d44cf33: Received BeginTabletCopySession request for tablet eb339ed8f9a141caa62cc5588d07fee1 from peer a8a749bf14364f22a4db0f7cdfa33ce9 ({username='slave'} at 127.22.242.67:40319)
I20240503 14:14:45.971762 5662 tablet_copy_service.cc:161] P e5237ac15b5c4a6c88a8a07f3d44cf33: Beginning new tablet copy session on tablet eb339ed8f9a141caa62cc5588d07fee1 from peer a8a749bf14364f22a4db0f7cdfa33ce9 at {username='slave'} at 127.22.242.67:40319: session id = a8a749bf14364f22a4db0f7cdfa33ce9-eb339ed8f9a141caa62cc5588d07fee1
I20240503 14:14:45.976652 5662 tablet_copy_source_session.cc:205] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:45.978830 7927 ts_tablet_manager.cc:1875] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:14:45.986559 7927 ts_tablet_manager.cc:1888] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 14:14:45.987015 7927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb339ed8f9a141caa62cc5588d07fee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:45.991362 7927 tablet_copy_client.cc:783] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:45.991817 7927 tablet_copy_client.cc:647] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:45.994282 7927 tablet_copy_client.cc:523] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:45.997697 7927 tablet_bootstrap.cc:492] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: Bootstrap starting.
I20240503 14:14:46.034689 7927 tablet_bootstrap.cc:492] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:46.035169 7927 tablet_bootstrap.cc:492] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: Bootstrap complete.
I20240503 14:14:46.035439 7927 ts_tablet_manager.cc:1367] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent bootstrapping tablet: real 0.038s user 0.033s sys 0.002s
I20240503 14:14:46.036141 7927 raft_consensus.cc:348] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } }
I20240503 14:14:46.036459 7927 raft_consensus.cc:729] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a8a749bf14364f22a4db0f7cdfa33ce9, State: Initialized, Role: LEARNER
I20240503 14:14:46.036685 7927 consensus_queue.cc:260] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } }
I20240503 14:14:46.037246 7927 ts_tablet_manager.cc:1398] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240503 14:14:46.038177 5662 tablet_copy_service.cc:342] P e5237ac15b5c4a6c88a8a07f3d44cf33: Request end of tablet copy session a8a749bf14364f22a4db0f7cdfa33ce9-eb339ed8f9a141caa62cc5588d07fee1 received from {username='slave'} at 127.22.242.67:40319
I20240503 14:14:46.038300 5662 tablet_copy_service.cc:433] P e5237ac15b5c4a6c88a8a07f3d44cf33: ending tablet copy session a8a749bf14364f22a4db0f7cdfa33ce9-eb339ed8f9a141caa62cc5588d07fee1 on tablet eb339ed8f9a141caa62cc5588d07fee1 with peer a8a749bf14364f22a4db0f7cdfa33ce9
I20240503 14:14:46.094808 7948 ts_tablet_manager.cc:902] T f6b542b55eb9477ab472b3a4005e6c63 P a24048f65ee54156b569d4bf1f738368: Initiating tablet copy from peer a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
I20240503 14:14:46.096387 7948 tablet_copy_client.cc:308] T f6b542b55eb9477ab472b3a4005e6c63 P a24048f65ee54156b569d4bf1f738368: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44189
I20240503 14:14:46.103260 5925 tablet_copy_service.cc:140] P a8a749bf14364f22a4db0f7cdfa33ce9: Received BeginTabletCopySession request for tablet f6b542b55eb9477ab472b3a4005e6c63 from peer a24048f65ee54156b569d4bf1f738368 ({username='slave'} at 127.22.242.69:59275)
I20240503 14:14:46.103462 5925 tablet_copy_service.cc:161] P a8a749bf14364f22a4db0f7cdfa33ce9: Beginning new tablet copy session on tablet f6b542b55eb9477ab472b3a4005e6c63 from peer a24048f65ee54156b569d4bf1f738368 at {username='slave'} at 127.22.242.69:59275: session id = a24048f65ee54156b569d4bf1f738368-f6b542b55eb9477ab472b3a4005e6c63
I20240503 14:14:46.106531 5925 tablet_copy_source_session.cc:205] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:46.108062 7948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b542b55eb9477ab472b3a4005e6c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:46.112577 7954 ts_tablet_manager.cc:902] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6: Initiating tablet copy from peer 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071)
I20240503 14:14:46.115235 7948 tablet_copy_client.cc:783] T f6b542b55eb9477ab472b3a4005e6c63 P a24048f65ee54156b569d4bf1f738368: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:46.115543 7948 tablet_copy_client.cc:647] T f6b542b55eb9477ab472b3a4005e6c63 P a24048f65ee54156b569d4bf1f738368: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:46.116565 7954 tablet_copy_client.cc:308] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:41071
I20240503 14:14:46.117314 5794 tablet_copy_service.cc:140] P 5361a07e89ad47298aabe717a4f2867f: Received BeginTabletCopySession request for tablet 614d4da74aa547a28013c97f5a1079fd from peer da201109e13f4b36b3fba7477454a4c6 ({username='slave'} at 127.22.242.68:52581)
I20240503 14:14:46.117491 5794 tablet_copy_service.cc:161] P 5361a07e89ad47298aabe717a4f2867f: Beginning new tablet copy session on tablet 614d4da74aa547a28013c97f5a1079fd from peer da201109e13f4b36b3fba7477454a4c6 at {username='slave'} at 127.22.242.68:52581: session id = da201109e13f4b36b3fba7477454a4c6-614d4da74aa547a28013c97f5a1079fd
I20240503 14:14:46.117630 7948 tablet_copy_client.cc:523] T f6b542b55eb9477ab472b3a4005e6c63 P a24048f65ee54156b569d4bf1f738368: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:46.119580 5794 tablet_copy_source_session.cc:205] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:46.120405 7948 tablet_bootstrap.cc:492] T f6b542b55eb9477ab472b3a4005e6c63 P a24048f65ee54156b569d4bf1f738368: Bootstrap starting.
I20240503 14:14:46.120954 7954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 614d4da74aa547a28013c97f5a1079fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:46.127530 7954 tablet_copy_client.cc:783] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:46.127822 7954 tablet_copy_client.cc:647] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:46.129678 7954 tablet_copy_client.cc:523] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:46.132508 7954 tablet_bootstrap.cc:492] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6: Bootstrap starting.
I20240503 14:14:46.155803 7948 tablet_bootstrap.cc:492] T f6b542b55eb9477ab472b3a4005e6c63 P a24048f65ee54156b569d4bf1f738368: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:46.156241 7948 tablet_bootstrap.cc:492] T f6b542b55eb9477ab472b3a4005e6c63 P a24048f65ee54156b569d4bf1f738368: Bootstrap complete.
I20240503 14:14:46.156489 7948 ts_tablet_manager.cc:1367] T f6b542b55eb9477ab472b3a4005e6c63 P a24048f65ee54156b569d4bf1f738368: Time spent bootstrapping tablet: real 0.036s user 0.036s sys 0.000s
I20240503 14:14:46.157124 7948 raft_consensus.cc:348] T f6b542b55eb9477ab472b3a4005e6c63 P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } }
I20240503 14:14:46.157351 7948 raft_consensus.cc:729] T f6b542b55eb9477ab472b3a4005e6c63 P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a24048f65ee54156b569d4bf1f738368, State: Initialized, Role: LEARNER
I20240503 14:14:46.157552 7948 consensus_queue.cc:260] T f6b542b55eb9477ab472b3a4005e6c63 P a24048f65ee54156b569d4bf1f738368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } }
I20240503 14:14:46.158392 7948 ts_tablet_manager.cc:1398] T f6b542b55eb9477ab472b3a4005e6c63 P a24048f65ee54156b569d4bf1f738368: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:46.159408 5925 tablet_copy_service.cc:342] P a8a749bf14364f22a4db0f7cdfa33ce9: Request end of tablet copy session a24048f65ee54156b569d4bf1f738368-f6b542b55eb9477ab472b3a4005e6c63 received from {username='slave'} at 127.22.242.69:59275
I20240503 14:14:46.159557 5925 tablet_copy_service.cc:433] P a8a749bf14364f22a4db0f7cdfa33ce9: ending tablet copy session a24048f65ee54156b569d4bf1f738368-f6b542b55eb9477ab472b3a4005e6c63 on tablet f6b542b55eb9477ab472b3a4005e6c63 with peer a24048f65ee54156b569d4bf1f738368
I20240503 14:14:46.166661 7954 tablet_bootstrap.cc:492] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:46.167059 7954 tablet_bootstrap.cc:492] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6: Bootstrap complete.
I20240503 14:14:46.167331 7954 ts_tablet_manager.cc:1367] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.003s
I20240503 14:14:46.167975 7954 raft_consensus.cc:348] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } }
I20240503 14:14:46.168241 7954 raft_consensus.cc:729] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da201109e13f4b36b3fba7477454a4c6, State: Initialized, Role: LEARNER
I20240503 14:14:46.168442 7954 consensus_queue.cc:260] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } }
I20240503 14:14:46.169157 7954 ts_tablet_manager.cc:1398] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6: Time spent starting tablet: real 0.002s user 0.001s sys 0.004s
I20240503 14:14:46.169939 5794 tablet_copy_service.cc:342] P 5361a07e89ad47298aabe717a4f2867f: Request end of tablet copy session da201109e13f4b36b3fba7477454a4c6-614d4da74aa547a28013c97f5a1079fd received from {username='slave'} at 127.22.242.68:52581
I20240503 14:14:46.170089 5794 tablet_copy_service.cc:433] P 5361a07e89ad47298aabe717a4f2867f: ending tablet copy session da201109e13f4b36b3fba7477454a4c6-614d4da74aa547a28013c97f5a1079fd on tablet 614d4da74aa547a28013c97f5a1079fd with peer da201109e13f4b36b3fba7477454a4c6
I20240503 14:14:46.173328 7963 ts_tablet_manager.cc:902] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: Initiating tablet copy from peer 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071)
I20240503 14:14:46.175055 7963 tablet_copy_client.cc:308] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:41071
I20240503 14:14:46.175809 5794 tablet_copy_service.cc:140] P 5361a07e89ad47298aabe717a4f2867f: Received BeginTabletCopySession request for tablet bfd4f9dbf8a946c6bb2c2b37210efe55 from peer e5237ac15b5c4a6c88a8a07f3d44cf33 ({username='slave'} at 127.22.242.65:54431)
I20240503 14:14:46.175971 5794 tablet_copy_service.cc:161] P 5361a07e89ad47298aabe717a4f2867f: Beginning new tablet copy session on tablet bfd4f9dbf8a946c6bb2c2b37210efe55 from peer e5237ac15b5c4a6c88a8a07f3d44cf33 at {username='slave'} at 127.22.242.65:54431: session id = e5237ac15b5c4a6c88a8a07f3d44cf33-bfd4f9dbf8a946c6bb2c2b37210efe55
I20240503 14:14:46.177985 5794 tablet_copy_source_session.cc:205] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:46.179317 7963 ts_tablet_manager.cc:1875] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:14:46.180532 7927 ts_tablet_manager.cc:902] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: Initiating tablet copy from peer 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071)
I20240503 14:14:46.184429 7963 ts_tablet_manager.cc:1888] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240503 14:14:46.184856 7963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd4f9dbf8a946c6bb2c2b37210efe55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:46.187242 7963 tablet_copy_client.cc:783] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:46.187609 7963 tablet_copy_client.cc:647] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:46.189786 7963 tablet_copy_client.cc:523] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:46.193231 7963 tablet_bootstrap.cc:492] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: Bootstrap starting.
I20240503 14:14:46.202558 7927 tablet_copy_client.cc:308] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:41071
I20240503 14:14:46.203634 5794 tablet_copy_service.cc:140] P 5361a07e89ad47298aabe717a4f2867f: Received BeginTabletCopySession request for tablet 614d4da74aa547a28013c97f5a1079fd from peer a8a749bf14364f22a4db0f7cdfa33ce9 ({username='slave'} at 127.22.242.67:43285)
I20240503 14:14:46.203862 5794 tablet_copy_service.cc:161] P 5361a07e89ad47298aabe717a4f2867f: Beginning new tablet copy session on tablet 614d4da74aa547a28013c97f5a1079fd from peer a8a749bf14364f22a4db0f7cdfa33ce9 at {username='slave'} at 127.22.242.67:43285: session id = a8a749bf14364f22a4db0f7cdfa33ce9-614d4da74aa547a28013c97f5a1079fd
I20240503 14:14:46.206537 7948 ts_tablet_manager.cc:902] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368: Initiating tablet copy from peer a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
I20240503 14:14:46.207401 5794 tablet_copy_source_session.cc:205] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:46.209159 7927 ts_tablet_manager.cc:1875] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:14:46.217854 7927 ts_tablet_manager.cc:1888] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 14:14:46.218327 7927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 614d4da74aa547a28013c97f5a1079fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:46.223274 7927 tablet_copy_client.cc:783] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:46.223678 7927 tablet_copy_client.cc:647] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:46.226074 7927 tablet_copy_client.cc:523] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:46.230964 7927 tablet_bootstrap.cc:492] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: Bootstrap starting.
I20240503 14:14:46.242501 7948 tablet_copy_client.cc:308] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44189
I20240503 14:14:46.244040 5925 tablet_copy_service.cc:140] P a8a749bf14364f22a4db0f7cdfa33ce9: Received BeginTabletCopySession request for tablet 098804415c3b4bafa53e58119712758f from peer a24048f65ee54156b569d4bf1f738368 ({username='slave'} at 127.22.242.69:59275)
I20240503 14:14:46.244235 5925 tablet_copy_service.cc:161] P a8a749bf14364f22a4db0f7cdfa33ce9: Beginning new tablet copy session on tablet 098804415c3b4bafa53e58119712758f from peer a24048f65ee54156b569d4bf1f738368 at {username='slave'} at 127.22.242.69:59275: session id = a24048f65ee54156b569d4bf1f738368-098804415c3b4bafa53e58119712758f
I20240503 14:14:46.246490 5925 tablet_copy_source_session.cc:205] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:46.247886 7948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 098804415c3b4bafa53e58119712758f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:46.254513 7948 tablet_copy_client.cc:783] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:46.254873 7948 tablet_copy_client.cc:647] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:46.256314 7993 ts_tablet_manager.cc:902] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: Initiating tablet copy from peer a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189)
I20240503 14:14:46.262253 6834 raft_consensus.cc:1203] T 52983271755542f090bb26d0675ec84e P 7bb84965b41b41fa9be23175c82e0a3a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:46.266573 7948 tablet_copy_client.cc:523] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:46.270301 7948 tablet_bootstrap.cc:492] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368: Bootstrap starting.
I20240503 14:14:46.274765 7963 tablet_bootstrap.cc:492] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:46.275279 7963 tablet_bootstrap.cc:492] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: Bootstrap complete.
I20240503 14:14:46.275627 7963 ts_tablet_manager.cc:1367] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent bootstrapping tablet: real 0.082s user 0.061s sys 0.000s
I20240503 14:14:46.276567 7963 raft_consensus.cc:348] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: NON_VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: true } }
I20240503 14:14:46.276938 7963 raft_consensus.cc:729] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e5237ac15b5c4a6c88a8a07f3d44cf33, State: Initialized, Role: LEARNER
I20240503 14:14:46.277217 7963 consensus_queue.cc:260] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: NON_VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: true } }
I20240503 14:14:46.278580 7963 ts_tablet_manager.cc:1398] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:46.281482 5794 tablet_copy_service.cc:342] P 5361a07e89ad47298aabe717a4f2867f: Request end of tablet copy session e5237ac15b5c4a6c88a8a07f3d44cf33-bfd4f9dbf8a946c6bb2c2b37210efe55 received from {username='slave'} at 127.22.242.65:54431
I20240503 14:14:46.281697 5794 tablet_copy_service.cc:433] P 5361a07e89ad47298aabe717a4f2867f: ending tablet copy session e5237ac15b5c4a6c88a8a07f3d44cf33-bfd4f9dbf8a946c6bb2c2b37210efe55 on tablet bfd4f9dbf8a946c6bb2c2b37210efe55 with peer e5237ac15b5c4a6c88a8a07f3d44cf33
I20240503 14:14:46.362315 5903 raft_consensus.cc:1203] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:46.365635 7927 tablet_bootstrap.cc:492] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:46.367163 7927 tablet_bootstrap.cc:492] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: Bootstrap complete.
I20240503 14:14:46.367584 7927 ts_tablet_manager.cc:1367] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent bootstrapping tablet: real 0.137s user 0.078s sys 0.008s
I20240503 14:14:46.368777 7927 raft_consensus.cc:348] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } }
I20240503 14:14:46.369518 7927 raft_consensus.cc:729] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a8a749bf14364f22a4db0f7cdfa33ce9, State: Initialized, Role: LEARNER
I20240503 14:14:46.369870 7927 consensus_queue.cc:260] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: true } }
I20240503 14:14:46.370677 7927 ts_tablet_manager.cc:1398] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:46.372129 5794 tablet_copy_service.cc:342] P 5361a07e89ad47298aabe717a4f2867f: Request end of tablet copy session a8a749bf14364f22a4db0f7cdfa33ce9-614d4da74aa547a28013c97f5a1079fd received from {username='slave'} at 127.22.242.67:43285
I20240503 14:14:46.372352 5794 tablet_copy_service.cc:433] P 5361a07e89ad47298aabe717a4f2867f: ending tablet copy session a8a749bf14364f22a4db0f7cdfa33ce9-614d4da74aa547a28013c97f5a1079fd on tablet 614d4da74aa547a28013c97f5a1079fd with peer a8a749bf14364f22a4db0f7cdfa33ce9
I20240503 14:14:46.376844 7993 tablet_copy_client.cc:308] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:44189
I20240503 14:14:46.378103 5925 tablet_copy_service.cc:140] P a8a749bf14364f22a4db0f7cdfa33ce9: Received BeginTabletCopySession request for tablet 098804415c3b4bafa53e58119712758f from peer 5361a07e89ad47298aabe717a4f2867f ({username='slave'} at 127.22.242.66:59155)
I20240503 14:14:46.378317 5925 tablet_copy_service.cc:161] P a8a749bf14364f22a4db0f7cdfa33ce9: Beginning new tablet copy session on tablet 098804415c3b4bafa53e58119712758f from peer 5361a07e89ad47298aabe717a4f2867f at {username='slave'} at 127.22.242.66:59155: session id = 5361a07e89ad47298aabe717a4f2867f-098804415c3b4bafa53e58119712758f
I20240503 14:14:46.382361 5925 tablet_copy_source_session.cc:205] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:46.384281 7993 ts_tablet_manager.cc:1875] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:14:46.391963 7993 ts_tablet_manager.cc:1888] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 14:14:46.392284 7993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 098804415c3b4bafa53e58119712758f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:46.395143 7993 tablet_copy_client.cc:783] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:46.395450 7993 tablet_copy_client.cc:647] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:46.395902 7948 tablet_bootstrap.cc:492] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:46.396451 7948 tablet_bootstrap.cc:492] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368: Bootstrap complete.
I20240503 14:14:46.396737 7948 ts_tablet_manager.cc:1367] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368: Time spent bootstrapping tablet: real 0.127s user 0.059s sys 0.012s
I20240503 14:14:46.397779 7993 tablet_copy_client.cc:523] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:46.397641 7948 raft_consensus.cc:348] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: NON_VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: true } }
I20240503 14:14:46.398370 7948 raft_consensus.cc:729] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a24048f65ee54156b569d4bf1f738368, State: Initialized, Role: LEARNER
I20240503 14:14:46.398633 7948 consensus_queue.cc:260] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: NON_VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: true } }
I20240503 14:14:46.399272 7948 ts_tablet_manager.cc:1398] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:14:46.400395 5925 tablet_copy_service.cc:342] P a8a749bf14364f22a4db0f7cdfa33ce9: Request end of tablet copy session a24048f65ee54156b569d4bf1f738368-098804415c3b4bafa53e58119712758f received from {username='slave'} at 127.22.242.69:59275
I20240503 14:14:46.400534 5925 tablet_copy_service.cc:433] P a8a749bf14364f22a4db0f7cdfa33ce9: ending tablet copy session a24048f65ee54156b569d4bf1f738368-098804415c3b4bafa53e58119712758f on tablet 098804415c3b4bafa53e58119712758f with peer a24048f65ee54156b569d4bf1f738368
I20240503 14:14:46.404824 7993 tablet_bootstrap.cc:492] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: Bootstrap starting.
I20240503 14:14:46.442413 7993 tablet_bootstrap.cc:492] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:46.442891 7993 tablet_bootstrap.cc:492] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: Bootstrap complete.
I20240503 14:14:46.443161 7993 ts_tablet_manager.cc:1367] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: Time spent bootstrapping tablet: real 0.038s user 0.032s sys 0.003s
I20240503 14:14:46.443876 7993 raft_consensus.cc:348] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: NON_VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: true } }
I20240503 14:14:46.444204 7993 raft_consensus.cc:729] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5361a07e89ad47298aabe717a4f2867f, State: Initialized, Role: LEARNER
I20240503 14:14:46.444428 7993 consensus_queue.cc:260] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: NON_VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: true } }
I20240503 14:14:46.444983 7993 ts_tablet_manager.cc:1398] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240503 14:14:46.445863 5925 tablet_copy_service.cc:342] P a8a749bf14364f22a4db0f7cdfa33ce9: Request end of tablet copy session 5361a07e89ad47298aabe717a4f2867f-098804415c3b4bafa53e58119712758f received from {username='slave'} at 127.22.242.66:59155
I20240503 14:14:46.446014 5925 tablet_copy_service.cc:433] P a8a749bf14364f22a4db0f7cdfa33ce9: ending tablet copy session 5361a07e89ad47298aabe717a4f2867f-098804415c3b4bafa53e58119712758f on tablet 098804415c3b4bafa53e58119712758f with peer 5361a07e89ad47298aabe717a4f2867f
I20240503 14:14:46.539201 6702 raft_consensus.cc:1203] T f6b542b55eb9477ab472b3a4005e6c63 P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:46.654544 6702 raft_consensus.cc:1203] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:14:46.657572 5903 raft_consensus.cc:1203] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:14:46.661137 5772 raft_consensus.cc:1203] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:14:46.720821 6964 raft_consensus.cc:1203] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:14:46.722754 5641 raft_consensus.cc:1203] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 14:14:46.780282 8014 raft_consensus.cc:1050] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33: attempting to promote NON_VOTER a8a749bf14364f22a4db0f7cdfa33ce9 to VOTER
I20240503 14:14:46.780933 8014 consensus_queue.cc:237] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } }
I20240503 14:14:46.783115 6702 raft_consensus.cc:1261] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:46.783135 5903 raft_consensus.cc:1261] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEARNER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:46.783164 5772 raft_consensus.cc:1261] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:46.783730 8007 consensus_queue.cc:1035] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:46.784078 8017 consensus_queue.cc:1035] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:46.784401 8014 consensus_queue.cc:1035] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:46.788007 8017 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } } }
I20240503 14:14:46.788766 5903 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } } }
I20240503 14:14:46.788868 6702 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } } }
I20240503 14:14:46.792536 5772 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } } }
I20240503 14:14:46.793926 7641 raft_consensus.cc:1050] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f: attempting to promote NON_VOTER 7bb84965b41b41fa9be23175c82e0a3a to VOTER
I20240503 14:14:46.794453 5497 catalog_manager.cc:5484] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 reported cstate change: config changed from index 7 to 8, a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:46.794868 7641 consensus_queue.cc:237] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } }
I20240503 14:14:46.802759 6702 raft_consensus.cc:1261] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:46.803128 6834 raft_consensus.cc:1261] T 52983271755542f090bb26d0675ec84e P 7bb84965b41b41fa9be23175c82e0a3a [term 1 LEARNER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:46.803725 8072 consensus_queue.cc:1035] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:46.804450 7641 consensus_queue.cc:1035] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:46.805701 5641 raft_consensus.cc:1261] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:46.806519 5642 consensus_queue.cc:237] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } }
I20240503 14:14:46.808720 5903 raft_consensus.cc:1261] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:46.809311 8017 consensus_queue.cc:1035] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:46.810639 6703 raft_consensus.cc:1261] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Refusing update from remote peer e5237ac15b5c4a6c88a8a07f3d44cf33: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:46.811411 8017 consensus_queue.cc:1035] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:46.813177 7694 consensus_queue.cc:1035] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:46.813436 8017 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } } }
I20240503 14:14:46.814134 5903 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } } }
I20240503 14:14:46.817839 6703 raft_consensus.cc:2932] T eb339ed8f9a141caa62cc5588d07fee1 P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } } }
I20240503 14:14:46.821282 5495 catalog_manager.cc:5484] T eb339ed8f9a141caa62cc5588d07fee1 P a8a749bf14364f22a4db0f7cdfa33ce9 reported cstate change: config changed from index 8 to 9, VOTER 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } } }
I20240503 14:14:46.823573 5485 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb339ed8f9a141caa62cc5588d07fee1 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:46.824016 7694 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:46.824591 5642 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:46.828754 6834 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:46.829442 5752 tablet_service.cc:1514] Processing DeleteTablet for tablet eb339ed8f9a141caa62cc5588d07fee1 with delete_type TABLET_DATA_TOMBSTONED (TS 5361a07e89ad47298aabe717a4f2867f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:46506
I20240503 14:14:46.832706 6703 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:46.833585 5494 catalog_manager.cc:5484] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f reported cstate change: config changed from index 7 to 8, 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:46.838732 5772 consensus_queue.cc:237] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } }
I20240503 14:14:46.839643 8086 tablet_replica.cc:330] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f: stopping tablet replica
I20240503 14:14:46.839857 8086 raft_consensus.cc:2229] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:46.840763 8086 raft_consensus.cc:2258] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:46.841228 5642 raft_consensus.cc:1261] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:46.841859 7694 consensus_queue.cc:1035] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:46.845674 5642 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:46.846030 6833 raft_consensus.cc:1261] T 52983271755542f090bb26d0675ec84e P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:46.846367 8085 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:46.848076 7801 consensus_queue.cc:1035] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:46.850200 8086 ts_tablet_manager.cc:1875] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:46.849857 6833 raft_consensus.cc:2932] T 52983271755542f090bb26d0675ec84e P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } } }
I20240503 14:14:46.851570 5487 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 52983271755542f090bb26d0675ec84e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:46.852507 5494 catalog_manager.cc:5484] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f reported cstate change: config changed from index 8 to 9, VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) evicted. New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:46.862872 8086 ts_tablet_manager.cc:1888] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:46.863142 8086 log.cc:1196] T eb339ed8f9a141caa62cc5588d07fee1 P 5361a07e89ad47298aabe717a4f2867f: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/eb339ed8f9a141caa62cc5588d07fee1
I20240503 14:14:46.864284 5487 catalog_manager.cc:4830] TS 5361a07e89ad47298aabe717a4f2867f (127.22.242.66:41071): tablet eb339ed8f9a141caa62cc5588d07fee1 (table rebalance_test_table_1 [id=a268a544233045a1b8bcb56727a9bbb8]) successfully deleted
I20240503 14:14:46.874938 6683 tablet_service.cc:1514] Processing DeleteTablet for tablet 52983271755542f090bb26d0675ec84e with delete_type TABLET_DATA_TOMBSTONED (TS a24048f65ee54156b569d4bf1f738368 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:42034
I20240503 14:14:46.876461 8099 tablet_replica.cc:330] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368: stopping tablet replica
I20240503 14:14:46.877136 8099 raft_consensus.cc:2229] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:46.877518 8099 raft_consensus.cc:2258] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:46.879030 8099 ts_tablet_manager.cc:1875] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:46.888262 8099 ts_tablet_manager.cc:1888] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:46.888543 8099 log.cc:1196] T 52983271755542f090bb26d0675ec84e P a24048f65ee54156b569d4bf1f738368: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal/wals/52983271755542f090bb26d0675ec84e
I20240503 14:14:46.889501 5488 catalog_manager.cc:4830] TS a24048f65ee54156b569d4bf1f738368 (127.22.242.69:42145): tablet 52983271755542f090bb26d0675ec84e (table rebalance_test_table_1 [id=a268a544233045a1b8bcb56727a9bbb8]) successfully deleted
I20240503 14:14:46.951053 8029 raft_consensus.cc:1050] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9: attempting to promote NON_VOTER a24048f65ee54156b569d4bf1f738368 to VOTER
I20240503 14:14:46.952291 8029 consensus_queue.cc:237] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } }
I20240503 14:14:46.955685 6964 raft_consensus.cc:1261] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:46.955926 6702 raft_consensus.cc:1261] T f6b542b55eb9477ab472b3a4005e6c63 P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:46.956864 8030 consensus_queue.cc:1035] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:46.957623 8030 consensus_queue.cc:1035] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:46.964020 5774 raft_consensus.cc:1261] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:46.964923 7817 consensus_queue.cc:1035] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:46.969105 5774 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:46.972615 7817 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:46.985617 5495 catalog_manager.cc:5484] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f reported cstate change: config changed from index 7 to 8, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:46.992412 6702 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:47.003268 5903 consensus_queue.cc:237] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } }
I20240503 14:14:47.014743 5774 raft_consensus.cc:1261] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:47.015712 8029 consensus_queue.cc:1035] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.022693 8030 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:47.027963 5485 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f6b542b55eb9477ab472b3a4005e6c63 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:47.029697 6701 raft_consensus.cc:1261] T f6b542b55eb9477ab472b3a4005e6c63 P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:47.029458 5772 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:47.031250 7817 consensus_queue.cc:1035] T f6b542b55eb9477ab472b3a4005e6c63 P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.033223 6701 raft_consensus.cc:2932] T f6b542b55eb9477ab472b3a4005e6c63 P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:47.034705 8029 raft_consensus.cc:1050] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9: attempting to promote NON_VOTER a24048f65ee54156b569d4bf1f738368 to VOTER
I20240503 14:14:47.035408 8029 consensus_queue.cc:237] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: NON_VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: true } }
I20240503 14:14:47.037942 5772 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 1 LEARNER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:47.038749 6703 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368 [term 1 LEARNER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:47.038652 8160 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: NON_VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.039166 6834 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:47.039436 8160 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.039949 8160 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.046447 5642 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:47.046379 8160 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: NON_VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: true } } }
I20240503 14:14:47.047103 8029 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.052570 6703 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: NON_VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: true } } }
I20240503 14:14:47.054561 5642 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: NON_VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: true } } }
I20240503 14:14:47.045204 5495 catalog_manager.cc:5484] T f6b542b55eb9477ab472b3a4005e6c63 P 5361a07e89ad47298aabe717a4f2867f reported cstate change: config changed from index 8 to 9, VOTER da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) evicted. New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } }
I20240503 14:14:47.060318 5774 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: NON_VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: true } } }
I20240503 14:14:47.062052 5497 catalog_manager.cc:5484] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 reported cstate change: config changed from index 8 to 9, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: NON_VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: true } } }
I20240503 14:14:47.066771 6834 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a24048f65ee54156b569d4bf1f738368 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: NON_VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: true } } }
I20240503 14:14:47.087450 8029 raft_consensus.cc:1050] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9: attempting to promote NON_VOTER 5361a07e89ad47298aabe717a4f2867f to VOTER
I20240503 14:14:47.087704 7817 raft_consensus.cc:1050] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9: attempting to promote NON_VOTER 5361a07e89ad47298aabe717a4f2867f to VOTER
I20240503 14:14:47.088706 8029 consensus_queue.cc:237] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } }
W20240503 14:14:47.091339 7817 raft_consensus.cc:1054] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9: Unable to promote non-voter 5361a07e89ad47298aabe717a4f2867f: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: NON_VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: true } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } }
I20240503 14:14:47.091375 6944 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b542b55eb9477ab472b3a4005e6c63 with delete_type TABLET_DATA_TOMBSTONED (TS da201109e13f4b36b3fba7477454a4c6 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51628
I20240503 14:14:47.092010 5642 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:14:47.092856 6703 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:14:47.093050 7817 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:47.093384 5774 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 1 LEARNER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:14:47.093796 6834 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:14:47.093997 8170 tablet_replica.cc:330] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6: stopping tablet replica
I20240503 14:14:47.094506 8170 raft_consensus.cc:2229] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:47.094847 8170 pending_rounds.cc:62] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6: Trying to abort 1 pending ops.
I20240503 14:14:47.094941 8170 pending_rounds.cc:68] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023598333753643008 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "f6b542b55eb9477ab472b3a4005e6c63" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } } }
I20240503 14:14:47.095398 7817 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:47.095615 8170 raft_consensus.cc:2894] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240503 14:14:47.095901 8170 raft_consensus.cc:2866] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 14:14:47.096118 8170 raft_consensus.cc:2258] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:47.096192 7817 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:47.096623 7817 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:47.098927 8167 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } } }
I20240503 14:14:47.099785 6834 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } } }
I20240503 14:14:47.102561 7694 raft_consensus.cc:1050] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f: attempting to promote NON_VOTER da201109e13f4b36b3fba7477454a4c6 to VOTER
I20240503 14:14:47.103524 7694 consensus_queue.cc:237] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } }
I20240503 14:14:47.104337 5774 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } } }
I20240503 14:14:47.104501 6701 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } } }
I20240503 14:14:47.110366 5494 catalog_manager.cc:5484] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368 reported cstate change: config changed from index 9 to 10, 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } } }
I20240503 14:14:47.112656 8170 ts_tablet_manager.cc:1875] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:47.114517 5642 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 5361a07e89ad47298aabe717a4f2867f (127.22.242.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "7bb84965b41b41fa9be23175c82e0a3a" member_type: VOTER last_known_addr { host: "127.22.242.70" port: 43863 } attrs { promote: false replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } } }
I20240503 14:14:47.119812 8170 ts_tablet_manager.cc:1888] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:47.119993 8170 log.cc:1196] T f6b542b55eb9477ab472b3a4005e6c63 P da201109e13f4b36b3fba7477454a4c6: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/wals/f6b542b55eb9477ab472b3a4005e6c63
I20240503 14:14:47.121264 5485 catalog_manager.cc:4830] TS da201109e13f4b36b3fba7477454a4c6 (127.22.242.68:42505): tablet f6b542b55eb9477ab472b3a4005e6c63 (table rebalance_test_table_0 [id=49c4626b38eb4b7194cd2add19c0cb32]) successfully deleted
I20240503 14:14:47.125706 5903 consensus_queue.cc:237] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } }
I20240503 14:14:47.128280 5642 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:14:47.128314 6703 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:14:47.128861 5774 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Refusing update from remote peer a8a749bf14364f22a4db0f7cdfa33ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:14:47.128882 7817 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:14:47.129185 8029 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:14:47.129463 8167 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:14:47.130571 6964 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6 [term 1 LEARNER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:47.131086 8085 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.133352 8171 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } } }
I20240503 14:14:47.134330 6703 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } } }
I20240503 14:14:47.139632 5774 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } } }
I20240503 14:14:47.141899 5485 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 098804415c3b4bafa53e58119712758f with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:47.141626 5642 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } } }
I20240503 14:14:47.145534 5494 catalog_manager.cc:5484] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f reported cstate change: config changed from index 10 to 11, VOTER 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70) evicted. New cstate: current_term: 1 leader_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } } }
I20240503 14:14:47.157850 6701 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:47.158236 5903 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEARNER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:47.158984 8085 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.159301 7694 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.164551 5642 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:47.165390 8085 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.166836 6814 tablet_service.cc:1514] Processing DeleteTablet for tablet 098804415c3b4bafa53e58119712758f with delete_type TABLET_DATA_TOMBSTONED (TS 7bb84965b41b41fa9be23175c82e0a3a not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:35238
I20240503 14:14:47.169050 7641 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.169683 5641 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.175071 6964 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.177333 8182 tablet_replica.cc:330] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a: stopping tablet replica
I20240503 14:14:47.178706 8182 raft_consensus.cc:2229] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:47.180428 8182 raft_consensus.cc:2258] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:47.180825 5494 catalog_manager.cc:5484] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 reported cstate change: config changed from index 8 to 9, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.188697 6701 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.192775 8182 ts_tablet_manager.cc:1875] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:47.197082 5774 consensus_queue.cc:237] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } }
I20240503 14:14:47.199191 6964 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:14:47.199693 7641 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:47.201742 6703 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:14:47.202558 8182 ts_tablet_manager.cc:1888] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 14:14:47.202279 7694 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.202874 8182 log.cc:1196] T 098804415c3b4bafa53e58119712758f P 7bb84965b41b41fa9be23175c82e0a3a: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-5/wal/wals/098804415c3b4bafa53e58119712758f
I20240503 14:14:47.203053 5904 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEARNER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:14:47.202793 6964 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.204074 7641 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.204120 5486 catalog_manager.cc:4830] TS 7bb84965b41b41fa9be23175c82e0a3a (127.22.242.70:43863): tablet 098804415c3b4bafa53e58119712758f (table rebalance_test_table_2 [id=632b93f2c4174b0080e62682cf1200b4]) successfully deleted
I20240503 14:14:47.205327 5904 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, da201109e13f4b36b3fba7477454a4c6 (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.206048 5487 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 614d4da74aa547a28013c97f5a1079fd with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:47.206447 8085 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:47.208091 5497 catalog_manager.cc:5484] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6 reported cstate change: config changed from index 9 to 10, VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.208726 6703 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.209283 5904 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEARNER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.212829 5620 tablet_service.cc:1514] Processing DeleteTablet for tablet 614d4da74aa547a28013c97f5a1079fd with delete_type TABLET_DATA_TOMBSTONED (TS e5237ac15b5c4a6c88a8a07f3d44cf33 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:33472
I20240503 14:14:47.216599 8085 raft_consensus.cc:1050] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f: attempting to promote NON_VOTER a8a749bf14364f22a4db0f7cdfa33ce9 to VOTER
I20240503 14:14:47.217422 8085 consensus_queue.cc:237] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } }
I20240503 14:14:47.218806 8184 tablet_replica.cc:330] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33: stopping tablet replica
I20240503 14:14:47.219053 8184 raft_consensus.cc:2229] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:47.219543 8184 raft_consensus.cc:2258] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:47.220077 5904 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEARNER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:14:47.221009 6703 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:14:47.221033 7641 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:14:47.221040 6964 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:14:47.221585 8085 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:14:47.221966 7694 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:14:47.224349 8184 ts_tablet_manager.cc:1875] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:47.225039 7694 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.230271 5904 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.231135 6701 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.234263 8184 ts_tablet_manager.cc:1888] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:14:47.235356 8184 log.cc:1196] T 614d4da74aa547a28013c97f5a1079fd P e5237ac15b5c4a6c88a8a07f3d44cf33: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/614d4da74aa547a28013c97f5a1079fd
I20240503 14:14:47.236349 5485 catalog_manager.cc:4830] TS e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157): tablet 614d4da74aa547a28013c97f5a1079fd (table rebalance_test_table_1 [id=a268a544233045a1b8bcb56727a9bbb8]) successfully deleted
I20240503 14:14:47.235658 6964 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.235922 5497 catalog_manager.cc:5484] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f reported cstate change: config changed from index 10 to 11, a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:47.246179 5774 consensus_queue.cc:237] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } }
I20240503 14:14:47.249480 5904 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:14:47.250056 6964 raft_consensus.cc:1261] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:14:47.250138 7694 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:14:47.251646 7694 consensus_queue.cc:1035] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:14:47.254029 8085 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P 5361a07e89ad47298aabe717a4f2867f [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.254575 5904 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.259922 5497 catalog_manager.cc:5484] T 614d4da74aa547a28013c97f5a1079fd P a8a749bf14364f22a4db0f7cdfa33ce9 reported cstate change: config changed from index 11 to 12, VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) evicted. New cstate: current_term: 1 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.260059 6964 raft_consensus.cc:2932] T 614d4da74aa547a28013c97f5a1079fd P da201109e13f4b36b3fba7477454a4c6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER a24048f65ee54156b569d4bf1f738368 (127.22.242.69) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { promote: false } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } }
I20240503 14:14:47.259943 5487 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 614d4da74aa547a28013c97f5a1079fd with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:47.264559 6683 tablet_service.cc:1514] Processing DeleteTablet for tablet 614d4da74aa547a28013c97f5a1079fd with delete_type TABLET_DATA_TOMBSTONED (TS a24048f65ee54156b569d4bf1f738368 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:42034
I20240503 14:14:47.264987 8099 tablet_replica.cc:330] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368: stopping tablet replica
I20240503 14:14:47.265177 8099 raft_consensus.cc:2229] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:47.265483 8099 raft_consensus.cc:2258] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:47.266335 8099 ts_tablet_manager.cc:1875] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:47.273900 8099 ts_tablet_manager.cc:1888] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240503 14:14:47.274194 8099 log.cc:1196] T 614d4da74aa547a28013c97f5a1079fd P a24048f65ee54156b569d4bf1f738368: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal/wals/614d4da74aa547a28013c97f5a1079fd
I20240503 14:14:47.275135 5488 catalog_manager.cc:4830] TS a24048f65ee54156b569d4bf1f738368 (127.22.242.69:42145): tablet 614d4da74aa547a28013c97f5a1079fd (table rebalance_test_table_1 [id=a268a544233045a1b8bcb56727a9bbb8]) successfully deleted
I20240503 14:14:47.292524 5904 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "098804415c3b4bafa53e58119712758f"
dest_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52566
I20240503 14:14:47.292832 5904 raft_consensus.cc:595] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:47.301311 7694 raft_consensus.cc:1050] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f: attempting to promote NON_VOTER e5237ac15b5c4a6c88a8a07f3d44cf33 to VOTER
I20240503 14:14:47.302055 7694 consensus_queue.cc:237] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: false } }
I20240503 14:14:47.304656 5641 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 2 LEARNER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:47.305101 6964 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6 [term 2 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:47.305688 5904 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 2 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:47.306335 8072 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.306866 8072 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.307368 8072 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.311071 8085 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: false } } }
I20240503 14:14:47.313349 5904 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: false } } }
I20240503 14:14:47.315140 6964 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: false } } }
I20240503 14:14:47.317622 5641 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: false } } }
I20240503 14:14:47.319190 5497 catalog_manager.cc:5484] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 reported cstate change: config changed from index 8 to 9, e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: false } } }
I20240503 14:14:47.326359 5774 consensus_queue.cc:237] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: false } }
I20240503 14:14:47.328884 5641 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 2 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:47.329042 6964 raft_consensus.cc:1261] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6 [term 2 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:47.329459 8085 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:47.330233 8085 consensus_queue.cc:1035] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:47.332331 8072 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P 5361a07e89ad47298aabe717a4f2867f [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: false } } }
I20240503 14:14:47.333386 6963 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: false } } }
I20240503 14:14:47.336539 5487 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bfd4f9dbf8a946c6bb2c2b37210efe55 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:47.336488 5641 raft_consensus.cc:2932] T bfd4f9dbf8a946c6bb2c2b37210efe55 P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: false } } }
I20240503 14:14:47.339094 5492 catalog_manager.cc:5484] T bfd4f9dbf8a946c6bb2c2b37210efe55 P da201109e13f4b36b3fba7477454a4c6 reported cstate change: config changed from index 9 to 10, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New cstate: current_term: 2 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } } peers { permanent_uuid: "da201109e13f4b36b3fba7477454a4c6" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 42505 } attrs { promote: false } } peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } attrs { promote: false } } }
I20240503 14:14:47.349188 5884 tablet_service.cc:1514] Processing DeleteTablet for tablet bfd4f9dbf8a946c6bb2c2b37210efe55 with delete_type TABLET_DATA_TOMBSTONED (TS a8a749bf14364f22a4db0f7cdfa33ce9 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:60268
I20240503 14:14:47.354399 8228 tablet_replica.cc:330] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9: stopping tablet replica
I20240503 14:14:47.355777 8228 raft_consensus.cc:2229] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:47.356487 8228 raft_consensus.cc:2258] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:47.358058 8228 ts_tablet_manager.cc:1875] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:47.369148 8228 ts_tablet_manager.cc:1888] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:14:47.369290 8228 log.cc:1196] T bfd4f9dbf8a946c6bb2c2b37210efe55 P a8a749bf14364f22a4db0f7cdfa33ce9: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/bfd4f9dbf8a946c6bb2c2b37210efe55
I20240503 14:14:47.370247 5485 catalog_manager.cc:4830] TS a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189): tablet bfd4f9dbf8a946c6bb2c2b37210efe55 (table rebalance_test_table_0 [id=49c4626b38eb4b7194cd2add19c0cb32]) successfully deleted
I20240503 14:14:47.382511 5904 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "098804415c3b4bafa53e58119712758f"
dest_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52590
I20240503 14:14:47.382743 5904 raft_consensus.cc:595] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:47.435657 5752 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:14:47.449831 6683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:47.472260 6814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:47.486550 6944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:47.495433 5884 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:47.528782 5620 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:47.539525 5904 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "098804415c3b4bafa53e58119712758f"
dest_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52610
I20240503 14:14:47.539777 5904 raft_consensus.cc:595] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:47.642500 8029 raft_consensus.cc:979] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9: : Instructing follower 5361a07e89ad47298aabe717a4f2867f to start an election
I20240503 14:14:47.642763 8029 raft_consensus.cc:1067] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Signalling peer 5361a07e89ad47298aabe717a4f2867f to start an election
I20240503 14:14:47.643571 5774 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "098804415c3b4bafa53e58119712758f"
dest_uuid: "5361a07e89ad47298aabe717a4f2867f"
from {username='slave'} at 127.22.242.67:43285
I20240503 14:14:47.643774 5774 raft_consensus.cc:482] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:47.643886 5774 raft_consensus.cc:3037] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:47.645802 5774 raft_consensus.cc:504] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } }
I20240503 14:14:47.646551 5774 leader_election.cc:288] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [CANDIDATE]: Term 2 election: Requested vote from peers e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157), a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189), a24048f65ee54156b569d4bf1f738368 (127.22.242.69:42145)
I20240503 14:14:47.646900 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098804415c3b4bafa53e58119712758f" candidate_uuid: "5361a07e89ad47298aabe717a4f2867f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33"
I20240503 14:14:47.647050 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098804415c3b4bafa53e58119712758f" candidate_uuid: "5361a07e89ad47298aabe717a4f2867f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9"
I20240503 14:14:47.647137 5641 raft_consensus.cc:3037] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:47.647276 5904 raft_consensus.cc:3032] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:47.647387 5904 raft_consensus.cc:729] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a8a749bf14364f22a4db0f7cdfa33ce9, State: Running, Role: LEADER
I20240503 14:14:47.647373 6701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098804415c3b4bafa53e58119712758f" candidate_uuid: "5361a07e89ad47298aabe717a4f2867f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "a24048f65ee54156b569d4bf1f738368"
I20240503 14:14:47.647670 6701 raft_consensus.cc:3037] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:47.647699 5904 consensus_queue.cc:260] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } }
I20240503 14:14:47.648142 5904 raft_consensus.cc:3037] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:47.649917 5904 raft_consensus.cc:2454] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5361a07e89ad47298aabe717a4f2867f in term 2.
I20240503 14:14:47.649950 5641 raft_consensus.cc:2454] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5361a07e89ad47298aabe717a4f2867f in term 2.
I20240503 14:14:47.650509 5707 leader_election.cc:302] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5361a07e89ad47298aabe717a4f2867f, a8a749bf14364f22a4db0f7cdfa33ce9, e5237ac15b5c4a6c88a8a07f3d44cf33; no voters:
I20240503 14:14:47.650628 6701 raft_consensus.cc:2454] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5361a07e89ad47298aabe717a4f2867f in term 2.
I20240503 14:14:47.650832 8072 raft_consensus.cc:2781] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:47.651022 8072 raft_consensus.cc:686] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 2 LEADER]: Becoming Leader. State: Replica: 5361a07e89ad47298aabe717a4f2867f, State: Running, Role: LEADER
I20240503 14:14:47.651330 8072 consensus_queue.cc:237] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } }
I20240503 14:14:47.654075 5494 catalog_manager.cc:5484] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f reported cstate change: term changed from 1 to 2, leader changed from a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) to 5361a07e89ad47298aabe717a4f2867f (127.22.242.66). New cstate: current_term: 2 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:48.056351 5641 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 2 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 14:14:48.056995 8072 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:14:48.148869 5904 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 2 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 14:14:48.149484 8072 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a749bf14364f22a4db0f7cdfa33ce9" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 44189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:14:48.154870 6701 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368 [term 2 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 14:14:48.156270 8290 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:14:48.576493 5774 consensus_queue.cc:237] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } }
I20240503 14:14:48.578274 6701 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368 [term 2 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 14:14:48.578351 5641 raft_consensus.cc:1261] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 2 FOLLOWER]: Refusing update from remote peer 5361a07e89ad47298aabe717a4f2867f: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 14:14:48.578860 8288 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:14:48.579138 8290 consensus_queue.cc:1035] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:14:48.581840 8290 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P 5361a07e89ad47298aabe717a4f2867f [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } } }
I20240503 14:14:48.582312 6701 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } } }
I20240503 14:14:48.582464 5641 raft_consensus.cc:2932] T 098804415c3b4bafa53e58119712758f P e5237ac15b5c4a6c88a8a07f3d44cf33 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } } }
I20240503 14:14:48.586750 5487 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 098804415c3b4bafa53e58119712758f with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:48.587960 5495 catalog_manager.cc:5484] T 098804415c3b4bafa53e58119712758f P a24048f65ee54156b569d4bf1f738368 reported cstate change: config changed from index 11 to 13, VOTER a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67) evicted. New cstate: current_term: 2 leader_uuid: "5361a07e89ad47298aabe717a4f2867f" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e5237ac15b5c4a6c88a8a07f3d44cf33" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36157 } } peers { permanent_uuid: "a24048f65ee54156b569d4bf1f738368" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 42145 } attrs { promote: false } } peers { permanent_uuid: "5361a07e89ad47298aabe717a4f2867f" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41071 } attrs { promote: false } } }
I20240503 14:14:48.593848 5884 tablet_service.cc:1514] Processing DeleteTablet for tablet 098804415c3b4bafa53e58119712758f with delete_type TABLET_DATA_TOMBSTONED (TS a8a749bf14364f22a4db0f7cdfa33ce9 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:60268
I20240503 14:14:48.601264 8316 tablet_replica.cc:330] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9: stopping tablet replica
I20240503 14:14:48.601459 8316 raft_consensus.cc:2229] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:48.601661 8316 raft_consensus.cc:2258] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:48.602641 8316 ts_tablet_manager.cc:1875] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:48.608151 8316 ts_tablet_manager.cc:1888] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240503 14:14:48.608306 8316 log.cc:1196] T 098804415c3b4bafa53e58119712758f P a8a749bf14364f22a4db0f7cdfa33ce9: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/098804415c3b4bafa53e58119712758f
I20240503 14:14:48.609050 5485 catalog_manager.cc:4830] TS a8a749bf14364f22a4db0f7cdfa33ce9 (127.22.242.67:44189): tablet 098804415c3b4bafa53e58119712758f (table rebalance_test_table_2 [id=632b93f2c4174b0080e62682cf1200b4]) successfully deleted
I20240503 14:14:48.692168 5752 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:14:48.718784 6683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:48.720454 6814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:48.724251 5884 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:48.740945 6944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:48.752405 5620 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:48.889678 5752 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:14:48.890762 6814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:48.911010 6683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:48.928376 5884 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:48.944056 5620 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:48.944983 6944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:49.181461 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 5560
I20240503 14:14:49.190059 5681 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:49.334895 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 5560
W20240503 14:14:49.363269 6767 connection.cc:535] server connection from 127.22.242.65:42483 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:49.363873 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 5692
W20240503 14:14:49.363878 5707 connection.cc:535] client connection to 127.22.242.65:36157 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:49.364140 5707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:49.366281 5707 consensus_peers.cc:485] T 52983271755542f090bb26d0675ec84e P 5361a07e89ad47298aabe717a4f2867f -> Peer e5237ac15b5c4a6c88a8a07f3d44cf33 (127.22.242.65:36157): Couldn't send request to peer e5237ac15b5c4a6c88a8a07f3d44cf33. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:36157: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:49.370239 5814 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:49.529918 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 5692
I20240503 14:14:49.557485 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 5822
I20240503 14:14:49.563849 5944 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:49.718587 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 5822
I20240503 14:14:49.745841 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 6882
I20240503 14:14:49.751667 7004 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:49.866405 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 6882
I20240503 14:14:49.887833 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 6621
I20240503 14:14:49.893874 6743 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:50.015590 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 6621
I20240503 14:14:50.037788 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 6752
I20240503 14:14:50.043686 6874 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:50.158984 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 6752
W20240503 14:14:50.179908 5485 connection.cc:535] server connection from 127.22.242.70:56181 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:50.180151 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 5470
I20240503 14:14:50.181244 5531 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:50.326319 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 5470
2024-05-03T14:14:50Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3 (25430 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2
I20240503 14:14:50.388392 23497 test_util.cc:268] Using random seed: 1472296846
I20240503 14:14:50.390458 23497 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:14:50.390528 23497 ts_itest-base.cc:113] --------------
I20240503 14:14:50.390586 23497 ts_itest-base.cc:114] 5 tablet servers
I20240503 14:14:50.390646 23497 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:14:50.390702 23497 ts_itest-base.cc:116] --------------
2024-05-03T14:14:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:14:50Z Disabled control of system clock
I20240503 14:14:50.437485 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.242.126:36935
--webserver_interface=127.22.242.126
--webserver_port=0
--builtin_ntp_servers=127.22.242.84:43635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.242.126:36935
--location_mapping_cmd=/tmp/dist-test-taskqfBHnl/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 14:14:50.771369 8402 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 14:14:50.771704 8402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:50.771840 8402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:50.771951 8402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:50.779489 8402 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:50.779639 8402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:14:50.779721 8402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:50.779808 8402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:14:50.779893 8402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:14:50.789351 8402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:43635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskqfBHnl/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.22.242.126:36935
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.242.126:36935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.242.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:50.790736 8402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:50.792512 8402 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:50.802959 8407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:50.803081 8408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:50.803561 8410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:50.803637 8402 server_base.cc:1034] running on GCE node
I20240503 14:14:50.804479 8402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:50.805902 8402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:50.807137 8402 hybrid_clock.cc:648] HybridClock initialized: now 1714745690807075 us; error 39 us; skew 500 ppm
I20240503 14:14:50.807464 8402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:50.809806 8402 webserver.cc:460] Webserver started at http://127.22.242.126:34217/ using document root <none> and password file <none>
I20240503 14:14:50.810520 8402 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:50.810662 8402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:50.810958 8402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:50.813624 8402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "30e2b88dac944256b17f4a3abc977895"
format_stamp: "Formatted at 2024-05-03 14:14:50 on dist-test-slave-r3pd"
I20240503 14:14:50.814410 8402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "30e2b88dac944256b17f4a3abc977895"
format_stamp: "Formatted at 2024-05-03 14:14:50 on dist-test-slave-r3pd"
I20240503 14:14:50.819634 8402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240503 14:14:50.823298 8416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:50.824822 8402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:50.825006 8402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
uuid: "30e2b88dac944256b17f4a3abc977895"
format_stamp: "Formatted at 2024-05-03 14:14:50 on dist-test-slave-r3pd"
I20240503 14:14:50.825192 8402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:50.862627 8402 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:50.863523 8402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:50.863821 8402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:50.882460 8402 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.126:36935
I20240503 14:14:50.882489 8467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.126:36935 every 8 connection(s)
I20240503 14:14:50.883963 8402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:14:50.886822 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 8402
I20240503 14:14:50.887033 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:14:50.888064 8468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:50.899768 8468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895: Bootstrap starting.
I20240503 14:14:50.905579 8468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:50.906584 8468 log.cc:824] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:50.909345 8468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895: No bootstrap required, opened a new log
I20240503 14:14:50.915411 8468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2b88dac944256b17f4a3abc977895" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 36935 } }
I20240503 14:14:50.915768 8468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:50.915872 8468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e2b88dac944256b17f4a3abc977895, State: Initialized, Role: FOLLOWER
I20240503 14:14:50.916514 8468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2b88dac944256b17f4a3abc977895" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 36935 } }
I20240503 14:14:50.916712 8468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:50.916837 8468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:50.916986 8468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:50.919014 8468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2b88dac944256b17f4a3abc977895" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 36935 } }
I20240503 14:14:50.919513 8468 leader_election.cc:302] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30e2b88dac944256b17f4a3abc977895; no voters:
I20240503 14:14:50.920115 8468 leader_election.cc:288] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:14:50.920316 8473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:50.920928 8473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895 [term 1 LEADER]: Becoming Leader. State: Replica: 30e2b88dac944256b17f4a3abc977895, State: Running, Role: LEADER
I20240503 14:14:50.921481 8473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2b88dac944256b17f4a3abc977895" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 36935 } }
I20240503 14:14:50.922302 8468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:14:50.924546 8474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30e2b88dac944256b17f4a3abc977895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2b88dac944256b17f4a3abc977895" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 36935 } } }
I20240503 14:14:50.924983 8474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895 [sys.catalog]: This master's current role is: LEADER
I20240503 14:14:50.925623 8475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30e2b88dac944256b17f4a3abc977895. Latest consensus state: current_term: 1 leader_uuid: "30e2b88dac944256b17f4a3abc977895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2b88dac944256b17f4a3abc977895" member_type: VOTER last_known_addr { host: "127.22.242.126" port: 36935 } } }
I20240503 14:14:50.926026 8475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895 [sys.catalog]: This master's current role is: LEADER
I20240503 14:14:50.927608 8482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:14:50.933135 8482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:14:50.941870 8482 catalog_manager.cc:1347] Generated new cluster ID: ce35482dd42847b99b0d20c114767629
I20240503 14:14:50.942009 8482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:14:50.967294 8482 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:14:50.968616 8482 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:14:50.978493 8482 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 30e2b88dac944256b17f4a3abc977895: Generated new TSK 0
I20240503 14:14:50.979537 8482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:14:50.992646 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.65:0
--local_ip_for_outbound_sockets=127.22.242.65
--webserver_interface=127.22.242.65
--webserver_port=0
--tserver_master_addrs=127.22.242.126:36935
--builtin_ntp_servers=127.22.242.84:43635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:14:51.328307 8492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:51.328599 8492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:51.328689 8492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:51.336208 8492 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:51.336329 8492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:51.336449 8492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.65
I20240503 14:14:51.345377 8492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:43635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.242.65
--webserver_port=0
--tserver_master_addrs=127.22.242.126:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.65
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:51.346689 8492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:51.348452 8492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:51.364439 8497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:51.365553 8498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:51.365576 8500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:51.365792 8492 server_base.cc:1034] running on GCE node
I20240503 14:14:51.366508 8492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:51.367496 8492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:51.368695 8492 hybrid_clock.cc:648] HybridClock initialized: now 1714745691368632 us; error 56 us; skew 500 ppm
I20240503 14:14:51.369007 8492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:51.371248 8492 webserver.cc:460] Webserver started at http://127.22.242.65:43545/ using document root <none> and password file <none>
I20240503 14:14:51.372005 8492 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:51.372119 8492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:51.372443 8492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:51.375038 8492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6aab696fc59c4784a0814eadd67ae186"
format_stamp: "Formatted at 2024-05-03 14:14:51 on dist-test-slave-r3pd"
I20240503 14:14:51.375821 8492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6aab696fc59c4784a0814eadd67ae186"
format_stamp: "Formatted at 2024-05-03 14:14:51 on dist-test-slave-r3pd"
I20240503 14:14:51.381125 8492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 14:14:51.384711 8506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:51.386185 8492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240503 14:14:51.386358 8492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6aab696fc59c4784a0814eadd67ae186"
format_stamp: "Formatted at 2024-05-03 14:14:51 on dist-test-slave-r3pd"
I20240503 14:14:51.386582 8492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:51.406315 8492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:51.407223 8492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:51.407500 8492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:51.409164 8492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:51.410840 8492 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:51.410943 8492 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:51.411067 8492 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:51.411140 8492 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:51.443341 8492 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.65:36379
I20240503 14:14:51.443387 8618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.65:36379 every 8 connection(s)
I20240503 14:14:51.444870 8492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:14:51.455076 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 8492
I20240503 14:14:51.455274 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:14:51.457682 8619 heartbeater.cc:347] Connected to a master server at 127.22.242.126:36935
I20240503 14:14:51.457996 8619 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:51.459003 8619 heartbeater.cc:511] Master 127.22.242.126:36935 requested a full tablet report, sending...
I20240503 14:14:51.459297 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.66:0
--local_ip_for_outbound_sockets=127.22.242.66
--webserver_interface=127.22.242.66
--webserver_port=0
--tserver_master_addrs=127.22.242.126:36935
--builtin_ntp_servers=127.22.242.84:43635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:14:51.506067 8432 ts_manager.cc:194] Registered new tserver with Master: 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379)
I20240503 14:14:51.508531 8432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.65:51685
W20240503 14:14:51.794728 8624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:51.795040 8624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:51.795126 8624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:51.802387 8624 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:51.802510 8624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:51.802625 8624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.66
I20240503 14:14:51.811688 8624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:43635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.242.66
--webserver_port=0
--tserver_master_addrs=127.22.242.126:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.66
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:51.812988 8624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:51.814671 8624 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:51.826462 8629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:51.826634 8630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:51.826646 8632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:51.827735 8624 server_base.cc:1034] running on GCE node
I20240503 14:14:51.828339 8624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:51.829262 8624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:51.830453 8624 hybrid_clock.cc:648] HybridClock initialized: now 1714745691830388 us; error 56 us; skew 500 ppm
I20240503 14:14:51.830750 8624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:51.832998 8624 webserver.cc:460] Webserver started at http://127.22.242.66:43523/ using document root <none> and password file <none>
I20240503 14:14:51.833684 8624 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:51.833777 8624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:51.834045 8624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:51.836575 8624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f156149e6e7947818ef96621896db45b"
format_stamp: "Formatted at 2024-05-03 14:14:51 on dist-test-slave-r3pd"
I20240503 14:14:51.837280 8624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f156149e6e7947818ef96621896db45b"
format_stamp: "Formatted at 2024-05-03 14:14:51 on dist-test-slave-r3pd"
I20240503 14:14:51.842377 8624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:14:51.845777 8638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:51.847077 8624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:14:51.847229 8624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f156149e6e7947818ef96621896db45b"
format_stamp: "Formatted at 2024-05-03 14:14:51 on dist-test-slave-r3pd"
I20240503 14:14:51.847453 8624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:51.867030 8624 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:51.867905 8624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:51.868249 8624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:51.869240 8624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:51.870700 8624 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:51.870801 8624 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:51.870918 8624 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:51.871001 8624 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:51.908192 8624 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.66:41441
I20240503 14:14:51.908255 8750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.66:41441 every 8 connection(s)
I20240503 14:14:51.909590 8624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:14:51.918715 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 8624
I20240503 14:14:51.918893 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:14:51.922564 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.67:0
--local_ip_for_outbound_sockets=127.22.242.67
--webserver_interface=127.22.242.67
--webserver_port=0
--tserver_master_addrs=127.22.242.126:36935
--builtin_ntp_servers=127.22.242.84:43635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:14:51.922866 8751 heartbeater.cc:347] Connected to a master server at 127.22.242.126:36935
I20240503 14:14:51.923224 8751 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:51.924185 8751 heartbeater.cc:511] Master 127.22.242.126:36935 requested a full tablet report, sending...
I20240503 14:14:51.990860 8432 ts_manager.cc:194] Registered new tserver with Master: f156149e6e7947818ef96621896db45b (127.22.242.66:41441)
I20240503 14:14:51.992251 8432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.66:60061
W20240503 14:14:52.283578 8755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:52.283900 8755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:52.284027 8755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:52.291844 8755 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:52.291996 8755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:52.292184 8755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.67
I20240503 14:14:52.301414 8755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:43635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.242.67
--webserver_port=0
--tserver_master_addrs=127.22.242.126:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.67
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:52.302716 8755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:52.304422 8755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:52.316131 8761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:52.316253 8763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:52.316458 8760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:52.317129 8755 server_base.cc:1034] running on GCE node
I20240503 14:14:52.317759 8755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:52.318724 8755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:52.319928 8755 hybrid_clock.cc:648] HybridClock initialized: now 1714745692319881 us; error 45 us; skew 500 ppm
I20240503 14:14:52.320276 8755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:52.322657 8755 webserver.cc:460] Webserver started at http://127.22.242.67:33551/ using document root <none> and password file <none>
I20240503 14:14:52.323400 8755 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:52.323514 8755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:52.323843 8755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:52.326524 8755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c8d8f771315b46ba84b289cc0189229c"
format_stamp: "Formatted at 2024-05-03 14:14:52 on dist-test-slave-r3pd"
I20240503 14:14:52.327296 8755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c8d8f771315b46ba84b289cc0189229c"
format_stamp: "Formatted at 2024-05-03 14:14:52 on dist-test-slave-r3pd"
I20240503 14:14:52.332491 8755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240503 14:14:52.335747 8769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:52.337129 8755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:52.337273 8755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c8d8f771315b46ba84b289cc0189229c"
format_stamp: "Formatted at 2024-05-03 14:14:52 on dist-test-slave-r3pd"
I20240503 14:14:52.337463 8755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:52.363746 8755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:52.364665 8755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:52.364928 8755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:52.365896 8755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:52.367401 8755 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:52.367497 8755 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:52.367610 8755 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:52.367681 8755 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:52.399893 8755 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.67:37569
I20240503 14:14:52.399952 8881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.67:37569 every 8 connection(s)
I20240503 14:14:52.401224 8755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:14:52.402297 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 8755
I20240503 14:14:52.402493 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:14:52.406172 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.68:0
--local_ip_for_outbound_sockets=127.22.242.68
--webserver_interface=127.22.242.68
--webserver_port=0
--tserver_master_addrs=127.22.242.126:36935
--builtin_ntp_servers=127.22.242.84:43635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:14:52.415633 8882 heartbeater.cc:347] Connected to a master server at 127.22.242.126:36935
I20240503 14:14:52.415985 8882 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:52.417017 8882 heartbeater.cc:511] Master 127.22.242.126:36935 requested a full tablet report, sending...
I20240503 14:14:52.479984 8432 ts_manager.cc:194] Registered new tserver with Master: c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:14:52.481011 8432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.67:36619
I20240503 14:14:52.511798 8619 heartbeater.cc:503] Master 127.22.242.126:36935 was elected leader, sending a full tablet report...
W20240503 14:14:52.764024 8885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:52.764374 8885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:52.764470 8885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:52.771821 8885 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:52.771943 8885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:52.772058 8885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.68
I20240503 14:14:52.781134 8885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:43635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.242.68
--webserver_port=0
--tserver_master_addrs=127.22.242.126:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.68
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:52.782389 8885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:52.784052 8885 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:52.795758 8891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:52.795781 8892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:52.795833 8894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:52.796691 8885 server_base.cc:1034] running on GCE node
I20240503 14:14:52.797271 8885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:52.798184 8885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:52.799376 8885 hybrid_clock.cc:648] HybridClock initialized: now 1714745692799330 us; error 38 us; skew 500 ppm
I20240503 14:14:52.799690 8885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:52.801962 8885 webserver.cc:460] Webserver started at http://127.22.242.68:42183/ using document root <none> and password file <none>
I20240503 14:14:52.802806 8885 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:52.802922 8885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:52.803244 8885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:52.805810 8885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "9920fbe0481e4ee6960a390ac14e9e5c"
format_stamp: "Formatted at 2024-05-03 14:14:52 on dist-test-slave-r3pd"
I20240503 14:14:52.806586 8885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "9920fbe0481e4ee6960a390ac14e9e5c"
format_stamp: "Formatted at 2024-05-03 14:14:52 on dist-test-slave-r3pd"
I20240503 14:14:52.811544 8885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240503 14:14:52.814855 8900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:52.816098 8885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240503 14:14:52.816298 8885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "9920fbe0481e4ee6960a390ac14e9e5c"
format_stamp: "Formatted at 2024-05-03 14:14:52 on dist-test-slave-r3pd"
I20240503 14:14:52.816483 8885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:52.845377 8885 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:52.846244 8885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:52.846510 8885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:52.847493 8885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:52.849017 8885 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:52.849123 8885 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:52.849236 8885 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:52.849310 8885 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:52.880875 8885 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.68:32901
I20240503 14:14:52.880942 9012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.68:32901 every 8 connection(s)
I20240503 14:14:52.882220 8885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:14:52.888334 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 8885
I20240503 14:14:52.888528 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:14:52.892321 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.69:0
--local_ip_for_outbound_sockets=127.22.242.69
--webserver_interface=127.22.242.69
--webserver_port=0
--tserver_master_addrs=127.22.242.126:36935
--builtin_ntp_servers=127.22.242.84:43635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:14:52.896831 9013 heartbeater.cc:347] Connected to a master server at 127.22.242.126:36935
I20240503 14:14:52.897166 9013 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:52.898183 9013 heartbeater.cc:511] Master 127.22.242.126:36935 requested a full tablet report, sending...
I20240503 14:14:52.951742 8432 ts_manager.cc:194] Registered new tserver with Master: 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68:32901)
I20240503 14:14:52.953321 8432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.68:39279
I20240503 14:14:52.995712 8751 heartbeater.cc:503] Master 127.22.242.126:36935 was elected leader, sending a full tablet report...
W20240503 14:14:53.233052 9016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:53.233340 9016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:53.233438 9016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:53.240867 9016 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:53.240988 9016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:53.241108 9016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.69
I20240503 14:14:53.250169 9016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:43635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.242.69
--webserver_port=0
--tserver_master_addrs=127.22.242.126:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.69
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:53.251457 9016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:53.253226 9016 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:53.264636 9022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:53.264706 9025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:53.264706 9023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:53.265399 9016 server_base.cc:1034] running on GCE node
I20240503 14:14:53.265966 9016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:53.266896 9016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:53.268086 9016 hybrid_clock.cc:648] HybridClock initialized: now 1714745693268042 us; error 47 us; skew 500 ppm
I20240503 14:14:53.268395 9016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:53.270488 9016 webserver.cc:460] Webserver started at http://127.22.242.69:43655/ using document root <none> and password file <none>
I20240503 14:14:53.271188 9016 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:53.271282 9016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:53.271540 9016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:53.274111 9016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "85dca8600f3d46b5a5c972c4b343ea09"
format_stamp: "Formatted at 2024-05-03 14:14:53 on dist-test-slave-r3pd"
I20240503 14:14:53.274811 9016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "85dca8600f3d46b5a5c972c4b343ea09"
format_stamp: "Formatted at 2024-05-03 14:14:53 on dist-test-slave-r3pd"
I20240503 14:14:53.279693 9016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20240503 14:14:53.282974 9031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:53.284293 9016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:53.284443 9016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "85dca8600f3d46b5a5c972c4b343ea09"
format_stamp: "Formatted at 2024-05-03 14:14:53 on dist-test-slave-r3pd"
I20240503 14:14:53.284641 9016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:53.302280 9016 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:53.303135 9016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:53.303422 9016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:53.304380 9016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:53.305831 9016 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:53.305918 9016 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:53.306000 9016 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:53.306056 9016 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:53.337209 9016 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.69:41037
I20240503 14:14:53.337284 9143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.69:41037 every 8 connection(s)
I20240503 14:14:53.338517 9016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:14:53.344250 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 9016
I20240503 14:14:53.344470 23497 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:14:53.351416 9144 heartbeater.cc:347] Connected to a master server at 127.22.242.126:36935
I20240503 14:14:53.351706 9144 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:53.352525 9144 heartbeater.cc:511] Master 127.22.242.126:36935 requested a full tablet report, sending...
I20240503 14:14:53.400667 8432 ts_manager.cc:194] Registered new tserver with Master: 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69:41037)
I20240503 14:14:53.401600 8432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.69:34871
I20240503 14:14:53.412580 23497 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 14:14:53.428789 23497 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 14:14:53.428973 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 9016
I20240503 14:14:53.437618 9139 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:53.484781 8882 heartbeater.cc:503] Master 127.22.242.126:36935 was elected leader, sending a full tablet report...
I20240503 14:14:53.517421 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 9016
I20240503 14:14:53.533778 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 8885
I20240503 14:14:53.542254 9008 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:53.621812 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 8885
I20240503 14:14:57.392515 8432 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51838:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:14:57.399116 8432 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:14:57.440187 8554 tablet_service.cc:1467] Processing CreateTablet for tablet e25a10096a2b4216a0f5714e23b4d643 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ee49efb58224c89a5d2831f003507e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:57.441695 8554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25a10096a2b4216a0f5714e23b4d643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:57.442216 8815 tablet_service.cc:1467] Processing CreateTablet for tablet a061e8fe907c4f5899d1d90cf1c06754 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ee49efb58224c89a5d2831f003507e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:57.442869 8817 tablet_service.cc:1467] Processing CreateTablet for tablet e25a10096a2b4216a0f5714e23b4d643 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ee49efb58224c89a5d2831f003507e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:57.442155 8816 tablet_service.cc:1467] Processing CreateTablet for tablet 24c71780b3154d54b50cf61ec123d6d5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ee49efb58224c89a5d2831f003507e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:57.443475 8815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a061e8fe907c4f5899d1d90cf1c06754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:57.443909 8816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c71780b3154d54b50cf61ec123d6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:57.444918 8817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25a10096a2b4216a0f5714e23b4d643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:57.440369 8552 tablet_service.cc:1467] Processing CreateTablet for tablet a061e8fe907c4f5899d1d90cf1c06754 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ee49efb58224c89a5d2831f003507e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:57.448212 8552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a061e8fe907c4f5899d1d90cf1c06754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:57.449381 8685 tablet_service.cc:1467] Processing CreateTablet for tablet 24c71780b3154d54b50cf61ec123d6d5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ee49efb58224c89a5d2831f003507e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:57.450672 8685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c71780b3154d54b50cf61ec123d6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:57.451023 8684 tablet_service.cc:1467] Processing CreateTablet for tablet a061e8fe907c4f5899d1d90cf1c06754 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ee49efb58224c89a5d2831f003507e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:57.451484 8684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a061e8fe907c4f5899d1d90cf1c06754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:57.460228 8553 tablet_service.cc:1467] Processing CreateTablet for tablet 24c71780b3154d54b50cf61ec123d6d5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ee49efb58224c89a5d2831f003507e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:57.460577 8686 tablet_service.cc:1467] Processing CreateTablet for tablet e25a10096a2b4216a0f5714e23b4d643 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ee49efb58224c89a5d2831f003507e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:57.461026 8686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25a10096a2b4216a0f5714e23b4d643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:57.461068 8553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c71780b3154d54b50cf61ec123d6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:57.473431 9170 tablet_bootstrap.cc:492] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b: Bootstrap starting.
I20240503 14:14:57.478482 9170 tablet_bootstrap.cc:654] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:57.479511 9169 tablet_bootstrap.cc:492] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186: Bootstrap starting.
I20240503 14:14:57.481922 9168 tablet_bootstrap.cc:492] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c: Bootstrap starting.
I20240503 14:14:57.483678 9169 tablet_bootstrap.cc:654] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:57.484818 9169 log.cc:824] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:57.485433 9168 tablet_bootstrap.cc:654] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:57.485936 9170 log.cc:824] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:57.486547 9168 log.cc:824] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:57.489002 9170 tablet_bootstrap.cc:492] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b: No bootstrap required, opened a new log
I20240503 14:14:57.489272 9170 ts_tablet_manager.cc:1367] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s
I20240503 14:14:57.489348 9169 tablet_bootstrap.cc:492] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186: No bootstrap required, opened a new log
I20240503 14:14:57.489348 9168 tablet_bootstrap.cc:492] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c: No bootstrap required, opened a new log
I20240503 14:14:57.489641 9168 ts_tablet_manager.cc:1367] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240503 14:14:57.489630 9169 ts_tablet_manager.cc:1367] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240503 14:14:57.496618 9170 raft_consensus.cc:348] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } }
I20240503 14:14:57.496996 9170 raft_consensus.cc:374] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:57.497121 9170 raft_consensus.cc:729] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f156149e6e7947818ef96621896db45b, State: Initialized, Role: FOLLOWER
I20240503 14:14:57.497766 9170 consensus_queue.cc:260] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } }
I20240503 14:14:57.498729 9169 raft_consensus.cc:348] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.499184 9169 raft_consensus.cc:374] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:57.499292 9169 raft_consensus.cc:729] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aab696fc59c4784a0814eadd67ae186, State: Initialized, Role: FOLLOWER
I20240503 14:14:57.499418 9170 ts_tablet_manager.cc:1398] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20240503 14:14:57.499850 9170 tablet_bootstrap.cc:492] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b: Bootstrap starting.
I20240503 14:14:57.499922 9169 consensus_queue.cc:260] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.500101 9168 raft_consensus.cc:348] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.500648 9168 raft_consensus.cc:374] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:57.500823 9168 raft_consensus.cc:729] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d8f771315b46ba84b289cc0189229c, State: Initialized, Role: FOLLOWER
I20240503 14:14:57.501533 9170 tablet_bootstrap.cc:654] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:57.501788 9168 consensus_queue.cc:260] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.505395 9170 tablet_bootstrap.cc:492] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b: No bootstrap required, opened a new log
I20240503 14:14:57.505613 9170 ts_tablet_manager.cc:1367] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240503 14:14:57.506436 9170 raft_consensus.cc:348] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.506600 9170 raft_consensus.cc:374] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:57.506729 9170 raft_consensus.cc:729] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f156149e6e7947818ef96621896db45b, State: Initialized, Role: FOLLOWER
I20240503 14:14:57.506948 9170 consensus_queue.cc:260] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.507323 9170 ts_tablet_manager.cc:1398] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:14:57.507591 9170 tablet_bootstrap.cc:492] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b: Bootstrap starting.
I20240503 14:14:57.509303 9170 tablet_bootstrap.cc:654] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:57.511840 9169 ts_tablet_manager.cc:1398] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186: Time spent starting tablet: real 0.022s user 0.017s sys 0.005s
I20240503 14:14:57.512363 9169 tablet_bootstrap.cc:492] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186: Bootstrap starting.
I20240503 14:14:57.513350 9168 ts_tablet_manager.cc:1398] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c: Time spent starting tablet: real 0.024s user 0.011s sys 0.008s
I20240503 14:14:57.513834 9168 tablet_bootstrap.cc:492] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c: Bootstrap starting.
I20240503 14:14:57.513983 9169 tablet_bootstrap.cc:654] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:57.514156 9170 tablet_bootstrap.cc:492] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b: No bootstrap required, opened a new log
I20240503 14:14:57.514379 9170 ts_tablet_manager.cc:1367] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:14:57.515120 9170 raft_consensus.cc:348] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.515410 9170 raft_consensus.cc:374] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:57.515553 9170 raft_consensus.cc:729] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f156149e6e7947818ef96621896db45b, State: Initialized, Role: FOLLOWER
I20240503 14:14:57.515820 9168 tablet_bootstrap.cc:654] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:57.515882 9170 consensus_queue.cc:260] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.516616 9170 ts_tablet_manager.cc:1398] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:14:57.518431 9168 tablet_bootstrap.cc:492] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c: No bootstrap required, opened a new log
I20240503 14:14:57.518569 9168 ts_tablet_manager.cc:1367] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:14:57.518579 9169 tablet_bootstrap.cc:492] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186: No bootstrap required, opened a new log
I20240503 14:14:57.518723 9169 ts_tablet_manager.cc:1367] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240503 14:14:57.519495 9169 raft_consensus.cc:348] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.519542 9168 raft_consensus.cc:348] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } }
I20240503 14:14:57.519707 9169 raft_consensus.cc:374] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:57.519748 9168 raft_consensus.cc:374] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:57.519798 9169 raft_consensus.cc:729] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aab696fc59c4784a0814eadd67ae186, State: Initialized, Role: FOLLOWER
I20240503 14:14:57.519837 9168 raft_consensus.cc:729] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d8f771315b46ba84b289cc0189229c, State: Initialized, Role: FOLLOWER
I20240503 14:14:57.520015 9169 consensus_queue.cc:260] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.520053 9168 consensus_queue.cc:260] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } }
I20240503 14:14:57.520664 9169 ts_tablet_manager.cc:1398] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:57.520953 9169 tablet_bootstrap.cc:492] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186: Bootstrap starting.
I20240503 14:14:57.521070 9168 ts_tablet_manager.cc:1398] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:57.521301 9174 raft_consensus.cc:482] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:57.521385 9168 tablet_bootstrap.cc:492] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c: Bootstrap starting.
I20240503 14:14:57.521536 9174 raft_consensus.cc:504] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.522311 9169 tablet_bootstrap.cc:654] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:57.523758 9174 leader_election.cc:288] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:14:57.524144 9168 tablet_bootstrap.cc:654] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:57.534106 9168 tablet_bootstrap.cc:492] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c: No bootstrap required, opened a new log
I20240503 14:14:57.534390 9168 ts_tablet_manager.cc:1367] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20240503 14:14:57.535481 9168 raft_consensus.cc:348] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.535861 9174 raft_consensus.cc:482] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:57.536002 9168 raft_consensus.cc:374] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:57.536116 9168 raft_consensus.cc:729] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d8f771315b46ba84b289cc0189229c, State: Initialized, Role: FOLLOWER
I20240503 14:14:57.536065 9174 raft_consensus.cc:504] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } }
I20240503 14:14:57.536489 9168 consensus_queue.cc:260] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.537576 9168 ts_tablet_manager.cc:1398] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:57.537986 9169 tablet_bootstrap.cc:492] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186: No bootstrap required, opened a new log
I20240503 14:14:57.538173 9169 ts_tablet_manager.cc:1367] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.004s
I20240503 14:14:57.538885 9169 raft_consensus.cc:348] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } }
I20240503 14:14:57.539170 9169 raft_consensus.cc:374] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:57.539271 9169 raft_consensus.cc:729] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aab696fc59c4784a0814eadd67ae186, State: Initialized, Role: FOLLOWER
I20240503 14:14:57.539517 9169 consensus_queue.cc:260] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } }
I20240503 14:14:57.540084 9169 ts_tablet_manager.cc:1398] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:14:57.540119 9176 raft_consensus.cc:482] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:57.540369 9176 raft_consensus.cc:504] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } }
I20240503 14:14:57.540589 9174 leader_election.cc:288] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:14:57.542341 9176 leader_election.cc:288] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379), f156149e6e7947818ef96621896db45b (127.22.242.66:41441)
I20240503 14:14:57.545006 8836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25a10096a2b4216a0f5714e23b4d643" candidate_uuid: "f156149e6e7947818ef96621896db45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8f771315b46ba84b289cc0189229c" is_pre_election: true
I20240503 14:14:57.545217 8574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a061e8fe907c4f5899d1d90cf1c06754" candidate_uuid: "f156149e6e7947818ef96621896db45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aab696fc59c4784a0814eadd67ae186" is_pre_election: true
I20240503 14:14:57.545248 8573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25a10096a2b4216a0f5714e23b4d643" candidate_uuid: "f156149e6e7947818ef96621896db45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aab696fc59c4784a0814eadd67ae186" is_pre_election: true
I20240503 14:14:57.545454 8836 raft_consensus.cc:2454] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f156149e6e7947818ef96621896db45b in term 0.
I20240503 14:14:57.545755 8573 raft_consensus.cc:2454] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f156149e6e7947818ef96621896db45b in term 0.
I20240503 14:14:57.545751 8574 raft_consensus.cc:2454] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f156149e6e7947818ef96621896db45b in term 0.
I20240503 14:14:57.545922 8837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a061e8fe907c4f5899d1d90cf1c06754" candidate_uuid: "f156149e6e7947818ef96621896db45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8f771315b46ba84b289cc0189229c" is_pre_election: true
I20240503 14:14:57.546113 8640 leader_election.cc:302] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d8f771315b46ba84b289cc0189229c, f156149e6e7947818ef96621896db45b; no voters:
I20240503 14:14:57.546286 8837 raft_consensus.cc:2454] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f156149e6e7947818ef96621896db45b in term 0.
I20240503 14:14:57.546602 9174 raft_consensus.cc:2781] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:57.546723 9174 raft_consensus.cc:482] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:57.546865 9174 raft_consensus.cc:3037] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:57.546918 8641 leader_election.cc:302] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aab696fc59c4784a0814eadd67ae186, f156149e6e7947818ef96621896db45b; no voters:
I20240503 14:14:57.549137 9188 raft_consensus.cc:2781] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:57.549264 9188 raft_consensus.cc:482] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:57.549412 9188 raft_consensus.cc:3037] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:57.551251 9174 raft_consensus.cc:504] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } }
I20240503 14:14:57.552595 8837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25a10096a2b4216a0f5714e23b4d643" candidate_uuid: "f156149e6e7947818ef96621896db45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8f771315b46ba84b289cc0189229c"
I20240503 14:14:57.552534 8573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25a10096a2b4216a0f5714e23b4d643" candidate_uuid: "c8d8f771315b46ba84b289cc0189229c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aab696fc59c4784a0814eadd67ae186" is_pre_election: true
I20240503 14:14:57.552788 8837 raft_consensus.cc:3037] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:57.552862 8573 raft_consensus.cc:2454] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d8f771315b46ba84b289cc0189229c in term 0.
I20240503 14:14:57.553282 8574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25a10096a2b4216a0f5714e23b4d643" candidate_uuid: "f156149e6e7947818ef96621896db45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aab696fc59c4784a0814eadd67ae186"
I20240503 14:14:57.553448 8574 raft_consensus.cc:3037] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:57.553767 9188 raft_consensus.cc:504] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.554508 9188 leader_election.cc:288] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [CANDIDATE]: Term 1 election: Requested vote from peers 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:14:57.555092 8837 raft_consensus.cc:2454] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f156149e6e7947818ef96621896db45b in term 1.
I20240503 14:14:57.555239 8573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a061e8fe907c4f5899d1d90cf1c06754" candidate_uuid: "f156149e6e7947818ef96621896db45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aab696fc59c4784a0814eadd67ae186"
I20240503 14:14:57.555415 8573 raft_consensus.cc:3037] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:57.555775 8772 leader_election.cc:302] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aab696fc59c4784a0814eadd67ae186, c8d8f771315b46ba84b289cc0189229c; no voters:
I20240503 14:14:57.556203 9176 raft_consensus.cc:2741] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 14:14:57.556614 8574 raft_consensus.cc:2454] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f156149e6e7947818ef96621896db45b in term 1.
I20240503 14:14:57.556771 8640 leader_election.cc:302] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d8f771315b46ba84b289cc0189229c, f156149e6e7947818ef96621896db45b; no voters:
I20240503 14:14:57.557116 8837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a061e8fe907c4f5899d1d90cf1c06754" candidate_uuid: "f156149e6e7947818ef96621896db45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8f771315b46ba84b289cc0189229c"
I20240503 14:14:57.557308 8837 raft_consensus.cc:3037] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:57.557413 9174 leader_election.cc:288] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [CANDIDATE]: Term 1 election: Requested vote from peers 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:14:57.557510 8706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25a10096a2b4216a0f5714e23b4d643" candidate_uuid: "c8d8f771315b46ba84b289cc0189229c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f156149e6e7947818ef96621896db45b" is_pre_election: true
I20240503 14:14:57.557830 9174 raft_consensus.cc:2781] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:57.558010 9174 raft_consensus.cc:686] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Becoming Leader. State: Replica: f156149e6e7947818ef96621896db45b, State: Running, Role: LEADER
I20240503 14:14:57.558203 8573 raft_consensus.cc:2454] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f156149e6e7947818ef96621896db45b in term 1.
I20240503 14:14:57.558627 8641 leader_election.cc:302] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aab696fc59c4784a0814eadd67ae186, f156149e6e7947818ef96621896db45b; no voters:
I20240503 14:14:57.558897 9188 raft_consensus.cc:2781] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:57.559059 9188 raft_consensus.cc:686] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Becoming Leader. State: Replica: f156149e6e7947818ef96621896db45b, State: Running, Role: LEADER
I20240503 14:14:57.558481 9174 consensus_queue.cc:237] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } }
I20240503 14:14:57.559435 9188 consensus_queue.cc:237] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.559681 8837 raft_consensus.cc:2454] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f156149e6e7947818ef96621896db45b in term 1.
I20240503 14:14:57.566339 8432 catalog_manager.cc:5484] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b reported cstate change: term changed from 0 to 1, leader changed from <none> to f156149e6e7947818ef96621896db45b (127.22.242.66). New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:57.571409 8432 catalog_manager.cc:5484] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b reported cstate change: term changed from 0 to 1, leader changed from <none> to f156149e6e7947818ef96621896db45b (127.22.242.66). New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: UNKNOWN } } }
W20240503 14:14:57.656759 8883 tablet.cc:2369] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:14:57.665329 8752 tablet.cc:2369] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:14:57.701500 8620 tablet.cc:2369] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:14:57.704694 9175 raft_consensus.cc:482] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:57.704885 9175 raft_consensus.cc:504] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.706457 9175 leader_election.cc:288] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f156149e6e7947818ef96621896db45b (127.22.242.66:41441), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:14:57.707231 9176 raft_consensus.cc:482] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:57.707404 9176 raft_consensus.cc:504] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.708122 9176 leader_election.cc:288] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379), f156149e6e7947818ef96621896db45b (127.22.242.66:41441)
I20240503 14:14:57.708703 8706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c71780b3154d54b50cf61ec123d6d5" candidate_uuid: "c8d8f771315b46ba84b289cc0189229c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f156149e6e7947818ef96621896db45b" is_pre_election: true
I20240503 14:14:57.708736 8573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c71780b3154d54b50cf61ec123d6d5" candidate_uuid: "c8d8f771315b46ba84b289cc0189229c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aab696fc59c4784a0814eadd67ae186" is_pre_election: true
I20240503 14:14:57.708966 8573 raft_consensus.cc:2454] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d8f771315b46ba84b289cc0189229c in term 0.
I20240503 14:14:57.708995 8706 raft_consensus.cc:2454] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d8f771315b46ba84b289cc0189229c in term 0.
I20240503 14:14:57.709405 8772 leader_election.cc:302] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aab696fc59c4784a0814eadd67ae186, c8d8f771315b46ba84b289cc0189229c; no voters:
I20240503 14:14:57.709669 9176 raft_consensus.cc:2781] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:57.709790 9176 raft_consensus.cc:482] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:57.709861 9176 raft_consensus.cc:3037] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:57.712419 9176 raft_consensus.cc:504] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.713091 9176 leader_election.cc:288] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [CANDIDATE]: Term 1 election: Requested vote from peers 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379), f156149e6e7947818ef96621896db45b (127.22.242.66:41441)
I20240503 14:14:57.713593 8706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c71780b3154d54b50cf61ec123d6d5" candidate_uuid: "c8d8f771315b46ba84b289cc0189229c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f156149e6e7947818ef96621896db45b"
I20240503 14:14:57.713850 8706 raft_consensus.cc:3037] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:57.713806 8573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c71780b3154d54b50cf61ec123d6d5" candidate_uuid: "c8d8f771315b46ba84b289cc0189229c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aab696fc59c4784a0814eadd67ae186"
I20240503 14:14:57.714022 8573 raft_consensus.cc:3037] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:57.714159 8705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c71780b3154d54b50cf61ec123d6d5" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f156149e6e7947818ef96621896db45b" is_pre_election: true
I20240503 14:14:57.716531 8837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c71780b3154d54b50cf61ec123d6d5" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8f771315b46ba84b289cc0189229c" is_pre_election: true
I20240503 14:14:57.716696 8573 raft_consensus.cc:2454] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d8f771315b46ba84b289cc0189229c in term 1.
I20240503 14:14:57.716755 8706 raft_consensus.cc:2454] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d8f771315b46ba84b289cc0189229c in term 1.
I20240503 14:14:57.716836 8837 raft_consensus.cc:2379] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6aab696fc59c4784a0814eadd67ae186 in current term 1: Already voted for candidate c8d8f771315b46ba84b289cc0189229c in this term.
I20240503 14:14:57.717113 8772 leader_election.cc:302] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aab696fc59c4784a0814eadd67ae186, c8d8f771315b46ba84b289cc0189229c; no voters:
I20240503 14:14:57.717435 9176 raft_consensus.cc:2781] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:57.717655 8508 leader_election.cc:302] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6aab696fc59c4784a0814eadd67ae186; no voters: c8d8f771315b46ba84b289cc0189229c, f156149e6e7947818ef96621896db45b
I20240503 14:14:57.717861 9176 raft_consensus.cc:686] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 1 LEADER]: Becoming Leader. State: Replica: c8d8f771315b46ba84b289cc0189229c, State: Running, Role: LEADER
I20240503 14:14:57.718089 9175 raft_consensus.cc:2726] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:14:57.718425 9176 consensus_queue.cc:237] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:57.722721 8432 catalog_manager.cc:5484] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d8f771315b46ba84b289cc0189229c (127.22.242.67). New cstate: current_term: 1 leader_uuid: "c8d8f771315b46ba84b289cc0189229c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:57.945392 9174 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:57.954789 9189 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:58.055733 9189 consensus_queue.cc:1035] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:58.060724 9189 consensus_queue.cc:1035] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:58.109476 9176 consensus_queue.cc:1035] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:58.114490 9176 consensus_queue.cc:1035] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:14:58.139569 9196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:58.139959 9196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:58.147557 9196 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 216.575 ms
time per row: 0.108288 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:14:59.442426 8430 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51838:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:14:59.443125 8430 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:14:59.452005 8554 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcba2de60f94132ab750ddc15b8afc5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9642b97573e44c3783fd3dddeffb9dfe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:59.452508 8554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcba2de60f94132ab750ddc15b8afc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:59.453325 8685 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcba2de60f94132ab750ddc15b8afc5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9642b97573e44c3783fd3dddeffb9dfe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:59.453732 8685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcba2de60f94132ab750ddc15b8afc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:59.454246 8816 tablet_service.cc:1467] Processing CreateTablet for tablet b311c3e036d34d3d8947978df14bae01 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9642b97573e44c3783fd3dddeffb9dfe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:59.454634 8816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b311c3e036d34d3d8947978df14bae01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:59.454607 8553 tablet_service.cc:1467] Processing CreateTablet for tablet b311c3e036d34d3d8947978df14bae01 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9642b97573e44c3783fd3dddeffb9dfe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:59.454968 8553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b311c3e036d34d3d8947978df14bae01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:59.455888 8815 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcba2de60f94132ab750ddc15b8afc5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9642b97573e44c3783fd3dddeffb9dfe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:59.456562 8552 tablet_service.cc:1467] Processing CreateTablet for tablet 326f8b8a3d754ff68d8f727d18318ff9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9642b97573e44c3783fd3dddeffb9dfe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:59.456573 8815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcba2de60f94132ab750ddc15b8afc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:59.456934 8552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 326f8b8a3d754ff68d8f727d18318ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:59.458926 8817 tablet_service.cc:1467] Processing CreateTablet for tablet 326f8b8a3d754ff68d8f727d18318ff9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9642b97573e44c3783fd3dddeffb9dfe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:59.459298 8817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 326f8b8a3d754ff68d8f727d18318ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:59.461442 8684 tablet_service.cc:1467] Processing CreateTablet for tablet b311c3e036d34d3d8947978df14bae01 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9642b97573e44c3783fd3dddeffb9dfe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:59.461825 8684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b311c3e036d34d3d8947978df14bae01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:59.467453 8686 tablet_service.cc:1467] Processing CreateTablet for tablet 326f8b8a3d754ff68d8f727d18318ff9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9642b97573e44c3783fd3dddeffb9dfe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:59.467906 8686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 326f8b8a3d754ff68d8f727d18318ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:59.468696 9258 tablet_bootstrap.cc:492] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186: Bootstrap starting.
I20240503 14:14:59.470269 9258 tablet_bootstrap.cc:654] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:59.471652 9260 tablet_bootstrap.cc:492] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b: Bootstrap starting.
I20240503 14:14:59.472669 9259 tablet_bootstrap.cc:492] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c: Bootstrap starting.
I20240503 14:14:59.473294 9260 tablet_bootstrap.cc:654] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:59.474224 9259 tablet_bootstrap.cc:654] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:59.474854 9258 tablet_bootstrap.cc:492] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186: No bootstrap required, opened a new log
I20240503 14:14:59.474956 9258 ts_tablet_manager.cc:1367] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:14:59.475638 9258 raft_consensus.cc:348] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.475831 9258 raft_consensus.cc:374] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:59.475896 9258 raft_consensus.cc:729] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aab696fc59c4784a0814eadd67ae186, State: Initialized, Role: FOLLOWER
I20240503 14:14:59.476094 9258 consensus_queue.cc:260] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.476388 9259 tablet_bootstrap.cc:492] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c: No bootstrap required, opened a new log
I20240503 14:14:59.476557 9259 ts_tablet_manager.cc:1367] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:14:59.477222 9260 tablet_bootstrap.cc:492] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b: No bootstrap required, opened a new log
I20240503 14:14:59.477293 9259 raft_consensus.cc:348] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.477429 9260 ts_tablet_manager.cc:1367] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 14:14:59.477547 9259 raft_consensus.cc:374] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:59.477690 9259 raft_consensus.cc:729] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d8f771315b46ba84b289cc0189229c, State: Initialized, Role: FOLLOWER
I20240503 14:14:59.477984 9259 consensus_queue.cc:260] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.478619 9259 ts_tablet_manager.cc:1398] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:14:59.478963 9259 tablet_bootstrap.cc:492] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c: Bootstrap starting.
I20240503 14:14:59.479103 9258 ts_tablet_manager.cc:1398] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:14:59.479394 9258 tablet_bootstrap.cc:492] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186: Bootstrap starting.
I20240503 14:14:59.480651 9259 tablet_bootstrap.cc:654] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:59.481074 9258 tablet_bootstrap.cc:654] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:59.478235 9260 raft_consensus.cc:348] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.482970 9260 raft_consensus.cc:374] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:59.483148 9260 raft_consensus.cc:729] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f156149e6e7947818ef96621896db45b, State: Initialized, Role: FOLLOWER
I20240503 14:14:59.483477 9260 consensus_queue.cc:260] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.484252 9260 ts_tablet_manager.cc:1398] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 14:14:59.484568 9260 tablet_bootstrap.cc:492] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b: Bootstrap starting.
I20240503 14:14:59.486258 9260 tablet_bootstrap.cc:654] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:59.487993 9258 tablet_bootstrap.cc:492] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186: No bootstrap required, opened a new log
I20240503 14:14:59.488196 9258 ts_tablet_manager.cc:1367] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20240503 14:14:59.488400 9260 tablet_bootstrap.cc:492] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b: No bootstrap required, opened a new log
I20240503 14:14:59.488556 9260 ts_tablet_manager.cc:1367] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:14:59.489130 9258 raft_consensus.cc:348] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.489228 9260 raft_consensus.cc:348] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.489384 9258 raft_consensus.cc:374] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:59.489480 9260 raft_consensus.cc:374] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:59.489478 9258 raft_consensus.cc:729] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aab696fc59c4784a0814eadd67ae186, State: Initialized, Role: FOLLOWER
I20240503 14:14:59.489554 9260 raft_consensus.cc:729] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f156149e6e7947818ef96621896db45b, State: Initialized, Role: FOLLOWER
I20240503 14:14:59.489732 9258 consensus_queue.cc:260] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.489780 9260 consensus_queue.cc:260] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.490270 9260 ts_tablet_manager.cc:1398] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:59.490551 9260 tablet_bootstrap.cc:492] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b: Bootstrap starting.
I20240503 14:14:59.490746 9258 ts_tablet_manager.cc:1398] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:14:59.490996 9258 tablet_bootstrap.cc:492] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186: Bootstrap starting.
I20240503 14:14:59.492076 9260 tablet_bootstrap.cc:654] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:59.492599 9258 tablet_bootstrap.cc:654] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:59.493522 9259 tablet_bootstrap.cc:492] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c: No bootstrap required, opened a new log
I20240503 14:14:59.493738 9259 ts_tablet_manager.cc:1367] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20240503 14:14:59.494724 9259 raft_consensus.cc:348] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.495038 9258 tablet_bootstrap.cc:492] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186: No bootstrap required, opened a new log
I20240503 14:14:59.495132 9259 raft_consensus.cc:374] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:59.495239 9258 ts_tablet_manager.cc:1367] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 14:14:59.495258 9259 raft_consensus.cc:729] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d8f771315b46ba84b289cc0189229c, State: Initialized, Role: FOLLOWER
I20240503 14:14:59.495513 9259 consensus_queue.cc:260] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.496127 9260 tablet_bootstrap.cc:492] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b: No bootstrap required, opened a new log
I20240503 14:14:59.496146 9259 ts_tablet_manager.cc:1398] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:59.496026 9258 raft_consensus.cc:348] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.496250 9258 raft_consensus.cc:374] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:59.496393 9260 ts_tablet_manager.cc:1367] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:14:59.496394 9258 raft_consensus.cc:729] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aab696fc59c4784a0814eadd67ae186, State: Initialized, Role: FOLLOWER
I20240503 14:14:59.496464 9259 tablet_bootstrap.cc:492] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c: Bootstrap starting.
I20240503 14:14:59.496640 9258 consensus_queue.cc:260] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.497301 9260 raft_consensus.cc:348] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.497553 9260 raft_consensus.cc:374] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:59.497699 9260 raft_consensus.cc:729] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f156149e6e7947818ef96621896db45b, State: Initialized, Role: FOLLOWER
I20240503 14:14:59.497859 9258 ts_tablet_manager.cc:1398] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:14:59.498034 9259 tablet_bootstrap.cc:654] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:59.498015 9260 consensus_queue.cc:260] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.499112 9260 ts_tablet_manager.cc:1398] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:59.500833 9259 tablet_bootstrap.cc:492] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c: No bootstrap required, opened a new log
I20240503 14:14:59.500977 9259 ts_tablet_manager.cc:1367] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:14:59.501776 9259 raft_consensus.cc:348] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.501981 9259 raft_consensus.cc:374] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:59.502110 9259 raft_consensus.cc:729] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d8f771315b46ba84b289cc0189229c, State: Initialized, Role: FOLLOWER
I20240503 14:14:59.502504 9259 consensus_queue.cc:260] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.502935 9259 ts_tablet_manager.cc:1398] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:14:59.504323 9264 raft_consensus.cc:482] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:59.504460 9264 raft_consensus.cc:504] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.505019 9264 leader_election.cc:288] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f156149e6e7947818ef96621896db45b (127.22.242.66:41441), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:14:59.505506 8706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b311c3e036d34d3d8947978df14bae01" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f156149e6e7947818ef96621896db45b" is_pre_election: true
I20240503 14:14:59.505748 8706 raft_consensus.cc:2454] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 0.
I20240503 14:14:59.505754 8836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b311c3e036d34d3d8947978df14bae01" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8f771315b46ba84b289cc0189229c" is_pre_election: true
I20240503 14:14:59.505935 8836 raft_consensus.cc:2454] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 0.
I20240503 14:14:59.506206 8508 leader_election.cc:302] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aab696fc59c4784a0814eadd67ae186, f156149e6e7947818ef96621896db45b; no voters:
I20240503 14:14:59.506474 9264 raft_consensus.cc:2781] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:59.506594 9264 raft_consensus.cc:482] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:59.506705 9264 raft_consensus.cc:3037] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:59.508893 9264 raft_consensus.cc:504] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.509387 9264 leader_election.cc:288] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 1 election: Requested vote from peers f156149e6e7947818ef96621896db45b (127.22.242.66:41441), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:14:59.509876 8836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b311c3e036d34d3d8947978df14bae01" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8f771315b46ba84b289cc0189229c"
I20240503 14:14:59.509841 8706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b311c3e036d34d3d8947978df14bae01" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f156149e6e7947818ef96621896db45b"
I20240503 14:14:59.510066 8706 raft_consensus.cc:3037] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:59.510066 8836 raft_consensus.cc:3037] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:59.512864 8706 raft_consensus.cc:2454] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 1.
I20240503 14:14:59.512864 8836 raft_consensus.cc:2454] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 1.
I20240503 14:14:59.513253 8508 leader_election.cc:302] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aab696fc59c4784a0814eadd67ae186, f156149e6e7947818ef96621896db45b; no voters:
I20240503 14:14:59.513521 9264 raft_consensus.cc:2781] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:59.513897 9264 raft_consensus.cc:686] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Becoming Leader. State: Replica: 6aab696fc59c4784a0814eadd67ae186, State: Running, Role: LEADER
I20240503 14:14:59.514317 9264 consensus_queue.cc:237] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.517650 8430 catalog_manager.cc:5484] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:59.623982 9264 raft_consensus.cc:482] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:59.624270 9264 raft_consensus.cc:504] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.624925 9264 leader_election.cc:288] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f156149e6e7947818ef96621896db45b (127.22.242.66:41441), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:14:59.625432 8706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcba2de60f94132ab750ddc15b8afc5" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f156149e6e7947818ef96621896db45b" is_pre_election: true
I20240503 14:14:59.625524 8836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcba2de60f94132ab750ddc15b8afc5" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8f771315b46ba84b289cc0189229c" is_pre_election: true
I20240503 14:14:59.625685 8706 raft_consensus.cc:2454] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 0.
I20240503 14:14:59.625753 8836 raft_consensus.cc:2454] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 0.
I20240503 14:14:59.626111 8508 leader_election.cc:302] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aab696fc59c4784a0814eadd67ae186, f156149e6e7947818ef96621896db45b; no voters:
I20240503 14:14:59.626441 9264 raft_consensus.cc:2781] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:59.626528 9264 raft_consensus.cc:482] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:59.626647 9264 raft_consensus.cc:3037] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:59.628759 9264 raft_consensus.cc:504] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.629354 9264 leader_election.cc:288] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 1 election: Requested vote from peers f156149e6e7947818ef96621896db45b (127.22.242.66:41441), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:14:59.629838 8836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcba2de60f94132ab750ddc15b8afc5" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8f771315b46ba84b289cc0189229c"
I20240503 14:14:59.629817 8706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcba2de60f94132ab750ddc15b8afc5" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f156149e6e7947818ef96621896db45b"
I20240503 14:14:59.630034 8706 raft_consensus.cc:3037] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:59.630040 8836 raft_consensus.cc:3037] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:59.631881 8836 raft_consensus.cc:2454] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 1.
I20240503 14:14:59.631976 8706 raft_consensus.cc:2454] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 1.
I20240503 14:14:59.632324 8508 leader_election.cc:302] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aab696fc59c4784a0814eadd67ae186, c8d8f771315b46ba84b289cc0189229c; no voters:
I20240503 14:14:59.632581 9264 raft_consensus.cc:2781] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:59.632750 9264 raft_consensus.cc:686] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Becoming Leader. State: Replica: 6aab696fc59c4784a0814eadd67ae186, State: Running, Role: LEADER
I20240503 14:14:59.633004 9264 consensus_queue.cc:237] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.635411 8430 catalog_manager.cc:5484] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:59.689165 9174 raft_consensus.cc:482] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:59.689386 9174 raft_consensus.cc:504] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.690018 9174 leader_election.cc:288] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:14:59.690574 8836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "326f8b8a3d754ff68d8f727d18318ff9" candidate_uuid: "f156149e6e7947818ef96621896db45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8f771315b46ba84b289cc0189229c" is_pre_election: true
I20240503 14:14:59.690579 8572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "326f8b8a3d754ff68d8f727d18318ff9" candidate_uuid: "f156149e6e7947818ef96621896db45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aab696fc59c4784a0814eadd67ae186" is_pre_election: true
I20240503 14:14:59.690807 8572 raft_consensus.cc:2454] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f156149e6e7947818ef96621896db45b in term 0.
I20240503 14:14:59.690793 8836 raft_consensus.cc:2454] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f156149e6e7947818ef96621896db45b in term 0.
I20240503 14:14:59.691215 8640 leader_election.cc:302] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d8f771315b46ba84b289cc0189229c, f156149e6e7947818ef96621896db45b; no voters:
I20240503 14:14:59.691478 9174 raft_consensus.cc:2781] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:59.691576 9174 raft_consensus.cc:482] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:59.691694 9174 raft_consensus.cc:3037] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:59.693640 9174 raft_consensus.cc:504] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.694209 9174 leader_election.cc:288] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [CANDIDATE]: Term 1 election: Requested vote from peers 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:14:59.694568 8572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "326f8b8a3d754ff68d8f727d18318ff9" candidate_uuid: "f156149e6e7947818ef96621896db45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aab696fc59c4784a0814eadd67ae186"
I20240503 14:14:59.694659 8836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "326f8b8a3d754ff68d8f727d18318ff9" candidate_uuid: "f156149e6e7947818ef96621896db45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8f771315b46ba84b289cc0189229c"
I20240503 14:14:59.694762 8572 raft_consensus.cc:3037] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:59.694844 8836 raft_consensus.cc:3037] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:59.696760 8836 raft_consensus.cc:2454] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f156149e6e7947818ef96621896db45b in term 1.
I20240503 14:14:59.696890 8572 raft_consensus.cc:2454] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f156149e6e7947818ef96621896db45b in term 1.
I20240503 14:14:59.697134 8640 leader_election.cc:302] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d8f771315b46ba84b289cc0189229c, f156149e6e7947818ef96621896db45b; no voters:
I20240503 14:14:59.697379 9174 raft_consensus.cc:2781] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:59.697786 9174 raft_consensus.cc:686] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Becoming Leader. State: Replica: f156149e6e7947818ef96621896db45b, State: Running, Role: LEADER
I20240503 14:14:59.698091 9174 consensus_queue.cc:237] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:14:59.700479 8430 catalog_manager.cc:5484] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b reported cstate change: term changed from 0 to 1, leader changed from <none> to f156149e6e7947818ef96621896db45b (127.22.242.66). New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:59.909196 9264 consensus_queue.cc:1035] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:59.917325 9271 consensus_queue.cc:1035] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:00.089629 9174 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:00.095988 9274 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:00.110270 9276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:00.110610 9276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:00.118177 9276 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:15:00.145743 9271 consensus_queue.cc:1035] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240503 14:15:00.151631 9271 consensus_queue.cc:1035] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 214.111 ms
time per row: 0.107056 ms
I20240503 14:15:00.495651 8430 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51838:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:15:00.496428 8430 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:15:00.503959 8554 tablet_service.cc:1467] Processing CreateTablet for tablet 029f0fef59a740f88a26ee36fa15907f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca152a2c2e694184b98c458650a677ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:00.504142 8684 tablet_service.cc:1467] Processing CreateTablet for tablet 029f0fef59a740f88a26ee36fa15907f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca152a2c2e694184b98c458650a677ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:00.504494 8554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 029f0fef59a740f88a26ee36fa15907f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:00.504591 8684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 029f0fef59a740f88a26ee36fa15907f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:00.504550 8817 tablet_service.cc:1467] Processing CreateTablet for tablet 029f0fef59a740f88a26ee36fa15907f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca152a2c2e694184b98c458650a677ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:00.504998 8817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 029f0fef59a740f88a26ee36fa15907f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:00.509235 8686 tablet_service.cc:1467] Processing CreateTablet for tablet 1af64bab380347dbaa5c38577bb6957c (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca152a2c2e694184b98c458650a677ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:00.509642 8686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1af64bab380347dbaa5c38577bb6957c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:00.509887 8815 tablet_service.cc:1467] Processing CreateTablet for tablet 1af64bab380347dbaa5c38577bb6957c (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca152a2c2e694184b98c458650a677ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:00.510282 8815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1af64bab380347dbaa5c38577bb6957c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:00.512516 8816 tablet_service.cc:1467] Processing CreateTablet for tablet 610307a610a4495382bf62b26477a5db (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca152a2c2e694184b98c458650a677ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:00.512864 8816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610307a610a4495382bf62b26477a5db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:00.516171 8553 tablet_service.cc:1467] Processing CreateTablet for tablet 1af64bab380347dbaa5c38577bb6957c (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca152a2c2e694184b98c458650a677ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:00.516618 8553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1af64bab380347dbaa5c38577bb6957c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:00.518237 9339 tablet_bootstrap.cc:492] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b: Bootstrap starting.
I20240503 14:15:00.518898 8685 tablet_service.cc:1467] Processing CreateTablet for tablet 610307a610a4495382bf62b26477a5db (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca152a2c2e694184b98c458650a677ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:00.519276 8685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610307a610a4495382bf62b26477a5db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:00.519539 9338 tablet_bootstrap.cc:492] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186: Bootstrap starting.
I20240503 14:15:00.520696 8552 tablet_service.cc:1467] Processing CreateTablet for tablet 610307a610a4495382bf62b26477a5db (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca152a2c2e694184b98c458650a677ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:00.521162 8552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610307a610a4495382bf62b26477a5db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:00.521543 9338 tablet_bootstrap.cc:654] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:00.522294 9339 tablet_bootstrap.cc:654] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:00.531613 9338 tablet_bootstrap.cc:492] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186: No bootstrap required, opened a new log
I20240503 14:15:00.531937 9338 ts_tablet_manager.cc:1367] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20240503 14:15:00.532356 9340 tablet_bootstrap.cc:492] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: Bootstrap starting.
I20240503 14:15:00.532514 9339 tablet_bootstrap.cc:492] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b: No bootstrap required, opened a new log
I20240503 14:15:00.532663 9339 ts_tablet_manager.cc:1367] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240503 14:15:00.532814 9338 raft_consensus.cc:348] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.533025 9338 raft_consensus.cc:374] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:00.533105 9338 raft_consensus.cc:729] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aab696fc59c4784a0814eadd67ae186, State: Initialized, Role: FOLLOWER
I20240503 14:15:00.533365 9338 consensus_queue.cc:260] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.533447 9339 raft_consensus.cc:348] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.533622 9339 raft_consensus.cc:374] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:00.533730 9339 raft_consensus.cc:729] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f156149e6e7947818ef96621896db45b, State: Initialized, Role: FOLLOWER
I20240503 14:15:00.534057 9340 tablet_bootstrap.cc:654] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:00.533977 9339 consensus_queue.cc:260] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.534582 9338 ts_tablet_manager.cc:1398] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:15:00.534833 9338 tablet_bootstrap.cc:492] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186: Bootstrap starting.
I20240503 14:15:00.535492 9339 ts_tablet_manager.cc:1398] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:15:00.535763 9339 tablet_bootstrap.cc:492] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b: Bootstrap starting.
I20240503 14:15:00.536347 9338 tablet_bootstrap.cc:654] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:00.537400 9339 tablet_bootstrap.cc:654] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:00.540848 9338 tablet_bootstrap.cc:492] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186: No bootstrap required, opened a new log
I20240503 14:15:00.541034 9338 ts_tablet_manager.cc:1367] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240503 14:15:00.541177 9340 tablet_bootstrap.cc:492] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: No bootstrap required, opened a new log
I20240503 14:15:00.541345 9340 ts_tablet_manager.cc:1367] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240503 14:15:00.541918 9339 tablet_bootstrap.cc:492] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b: No bootstrap required, opened a new log
I20240503 14:15:00.541872 9338 raft_consensus.cc:348] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.542075 9339 ts_tablet_manager.cc:1367] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240503 14:15:00.542104 9338 raft_consensus.cc:374] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:00.542182 9338 raft_consensus.cc:729] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aab696fc59c4784a0814eadd67ae186, State: Initialized, Role: FOLLOWER
I20240503 14:15:00.542203 9340 raft_consensus.cc:348] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.542409 9340 raft_consensus.cc:374] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:00.542496 9340 raft_consensus.cc:729] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d8f771315b46ba84b289cc0189229c, State: Initialized, Role: FOLLOWER
I20240503 14:15:00.542433 9338 consensus_queue.cc:260] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.542775 9340 consensus_queue.cc:260] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.542927 9338 ts_tablet_manager.cc:1398] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:15:00.542954 9339 raft_consensus.cc:348] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.543192 9339 raft_consensus.cc:374] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:00.543247 9338 tablet_bootstrap.cc:492] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186: Bootstrap starting.
I20240503 14:15:00.543263 9339 raft_consensus.cc:729] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f156149e6e7947818ef96621896db45b, State: Initialized, Role: FOLLOWER
I20240503 14:15:00.543525 9339 consensus_queue.cc:260] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.543937 9340 ts_tablet_manager.cc:1398] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:00.544194 9339 ts_tablet_manager.cc:1398] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:15:00.544234 9340 tablet_bootstrap.cc:492] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c: Bootstrap starting.
I20240503 14:15:00.544481 9339 tablet_bootstrap.cc:492] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b: Bootstrap starting.
I20240503 14:15:00.545873 9340 tablet_bootstrap.cc:654] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:00.545945 9338 tablet_bootstrap.cc:654] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:00.546082 9339 tablet_bootstrap.cc:654] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:00.554776 9339 tablet_bootstrap.cc:492] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b: No bootstrap required, opened a new log
I20240503 14:15:00.554970 9339 ts_tablet_manager.cc:1367] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240503 14:15:00.555616 9340 tablet_bootstrap.cc:492] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c: No bootstrap required, opened a new log
I20240503 14:15:00.555795 9340 ts_tablet_manager.cc:1367] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.005s
I20240503 14:15:00.555820 9339 raft_consensus.cc:348] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.556013 9339 raft_consensus.cc:374] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:00.556109 9339 raft_consensus.cc:729] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f156149e6e7947818ef96621896db45b, State: Initialized, Role: FOLLOWER
I20240503 14:15:00.556142 9338 tablet_bootstrap.cc:492] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186: No bootstrap required, opened a new log
I20240503 14:15:00.556304 9338 ts_tablet_manager.cc:1367] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20240503 14:15:00.556376 9339 consensus_queue.cc:260] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.556603 9340 raft_consensus.cc:348] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.556829 9340 raft_consensus.cc:374] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:00.556912 9340 raft_consensus.cc:729] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d8f771315b46ba84b289cc0189229c, State: Initialized, Role: FOLLOWER
I20240503 14:15:00.556933 9339 ts_tablet_manager.cc:1398] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:15:00.557152 9340 consensus_queue.cc:260] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.557683 9338 raft_consensus.cc:348] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.557879 9338 raft_consensus.cc:374] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:00.557989 9338 raft_consensus.cc:729] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aab696fc59c4784a0814eadd67ae186, State: Initialized, Role: FOLLOWER
I20240503 14:15:00.558238 9338 consensus_queue.cc:260] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.559403 9338 ts_tablet_manager.cc:1398] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:15:00.559792 9340 ts_tablet_manager.cc:1398] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:15:00.560103 9340 tablet_bootstrap.cc:492] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c: Bootstrap starting.
I20240503 14:15:00.561812 9340 tablet_bootstrap.cc:654] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:00.563666 9340 tablet_bootstrap.cc:492] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c: No bootstrap required, opened a new log
I20240503 14:15:00.563783 9340 ts_tablet_manager.cc:1367] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:15:00.564579 9340 raft_consensus.cc:348] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.564786 9340 raft_consensus.cc:374] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:00.564888 9340 raft_consensus.cc:729] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d8f771315b46ba84b289cc0189229c, State: Initialized, Role: FOLLOWER
I20240503 14:15:00.565104 9340 consensus_queue.cc:260] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.565646 9340 ts_tablet_manager.cc:1398] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:15:00.671818 9319 raft_consensus.cc:482] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:00.672003 9319 raft_consensus.cc:504] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.672715 9319 leader_election.cc:288] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f156149e6e7947818ef96621896db45b (127.22.242.66:41441), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:15:00.673218 8705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "029f0fef59a740f88a26ee36fa15907f" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f156149e6e7947818ef96621896db45b" is_pre_election: true
I20240503 14:15:00.673355 8837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "029f0fef59a740f88a26ee36fa15907f" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8f771315b46ba84b289cc0189229c" is_pre_election: true
I20240503 14:15:00.673435 8705 raft_consensus.cc:2454] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 0.
I20240503 14:15:00.673575 8837 raft_consensus.cc:2454] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 0.
I20240503 14:15:00.673866 8508 leader_election.cc:302] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aab696fc59c4784a0814eadd67ae186, f156149e6e7947818ef96621896db45b; no voters:
I20240503 14:15:00.674146 9319 raft_consensus.cc:2781] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:00.674245 9319 raft_consensus.cc:482] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:00.674355 9319 raft_consensus.cc:3037] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:00.675132 9323 raft_consensus.cc:482] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:00.675272 9323 raft_consensus.cc:504] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.675921 9323 leader_election.cc:288] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f156149e6e7947818ef96621896db45b (127.22.242.66:41441), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:15:00.676285 8705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610307a610a4495382bf62b26477a5db" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f156149e6e7947818ef96621896db45b" is_pre_election: true
I20240503 14:15:00.676395 8837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610307a610a4495382bf62b26477a5db" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8f771315b46ba84b289cc0189229c" is_pre_election: true
I20240503 14:15:00.676491 8705 raft_consensus.cc:2454] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 0.
I20240503 14:15:00.676609 8837 raft_consensus.cc:2454] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 0.
I20240503 14:15:00.676810 9319 raft_consensus.cc:504] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.676906 8508 leader_election.cc:302] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aab696fc59c4784a0814eadd67ae186, f156149e6e7947818ef96621896db45b; no voters:
I20240503 14:15:00.677151 9323 raft_consensus.cc:2781] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:00.677268 9323 raft_consensus.cc:482] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:00.677368 9323 raft_consensus.cc:3037] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:00.677423 9319 leader_election.cc:288] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 1 election: Requested vote from peers f156149e6e7947818ef96621896db45b (127.22.242.66:41441), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:15:00.677724 8705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "029f0fef59a740f88a26ee36fa15907f" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f156149e6e7947818ef96621896db45b"
I20240503 14:15:00.677868 8837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "029f0fef59a740f88a26ee36fa15907f" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8f771315b46ba84b289cc0189229c"
I20240503 14:15:00.677939 8705 raft_consensus.cc:3037] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:00.678053 8837 raft_consensus.cc:3037] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:00.679376 9323 raft_consensus.cc:504] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.679920 9323 leader_election.cc:288] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 1 election: Requested vote from peers f156149e6e7947818ef96621896db45b (127.22.242.66:41441), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:15:00.680334 8836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610307a610a4495382bf62b26477a5db" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8f771315b46ba84b289cc0189229c"
I20240503 14:15:00.680513 8836 raft_consensus.cc:3037] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:00.680699 8705 raft_consensus.cc:2454] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 1.
I20240503 14:15:00.680790 8837 raft_consensus.cc:2454] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 1.
I20240503 14:15:00.681157 8508 leader_election.cc:302] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aab696fc59c4784a0814eadd67ae186, f156149e6e7947818ef96621896db45b; no voters:
I20240503 14:15:00.681411 9323 raft_consensus.cc:2781] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:00.681421 8705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610307a610a4495382bf62b26477a5db" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f156149e6e7947818ef96621896db45b"
I20240503 14:15:00.681541 9323 raft_consensus.cc:686] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Becoming Leader. State: Replica: 6aab696fc59c4784a0814eadd67ae186, State: Running, Role: LEADER
I20240503 14:15:00.681591 8705 raft_consensus.cc:3037] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:00.681769 9323 consensus_queue.cc:237] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.683058 8836 raft_consensus.cc:2454] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 1.
I20240503 14:15:00.683481 8508 leader_election.cc:302] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aab696fc59c4784a0814eadd67ae186, c8d8f771315b46ba84b289cc0189229c; no voters:
I20240503 14:15:00.683923 9323 raft_consensus.cc:2781] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:00.684077 9323 raft_consensus.cc:686] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Becoming Leader. State: Replica: 6aab696fc59c4784a0814eadd67ae186, State: Running, Role: LEADER
I20240503 14:15:00.684360 9323 consensus_queue.cc:237] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.684648 8705 raft_consensus.cc:2454] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 1.
I20240503 14:15:00.685045 8424 catalog_manager.cc:5484] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:00.689406 8424 catalog_manager.cc:5484] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65). New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:00.784363 9332 raft_consensus.cc:482] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:00.784557 9332 raft_consensus.cc:504] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.785198 9332 leader_election.cc:288] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:15:00.785676 8572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af64bab380347dbaa5c38577bb6957c" candidate_uuid: "f156149e6e7947818ef96621896db45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aab696fc59c4784a0814eadd67ae186" is_pre_election: true
I20240503 14:15:00.785754 8836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af64bab380347dbaa5c38577bb6957c" candidate_uuid: "f156149e6e7947818ef96621896db45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8f771315b46ba84b289cc0189229c" is_pre_election: true
I20240503 14:15:00.785944 8836 raft_consensus.cc:2454] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f156149e6e7947818ef96621896db45b in term 0.
I20240503 14:15:00.785935 8572 raft_consensus.cc:2454] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f156149e6e7947818ef96621896db45b in term 0.
I20240503 14:15:00.786360 8640 leader_election.cc:302] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d8f771315b46ba84b289cc0189229c, f156149e6e7947818ef96621896db45b; no voters:
I20240503 14:15:00.786619 9332 raft_consensus.cc:2781] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:00.786717 9332 raft_consensus.cc:482] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:00.786782 9332 raft_consensus.cc:3037] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:00.788849 9332 raft_consensus.cc:504] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.789399 9332 leader_election.cc:288] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [CANDIDATE]: Term 1 election: Requested vote from peers 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:15:00.789800 8572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af64bab380347dbaa5c38577bb6957c" candidate_uuid: "f156149e6e7947818ef96621896db45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aab696fc59c4784a0814eadd67ae186"
I20240503 14:15:00.789886 8836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af64bab380347dbaa5c38577bb6957c" candidate_uuid: "f156149e6e7947818ef96621896db45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8f771315b46ba84b289cc0189229c"
I20240503 14:15:00.790014 8572 raft_consensus.cc:3037] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:00.790078 8836 raft_consensus.cc:3037] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:00.792775 8572 raft_consensus.cc:2454] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f156149e6e7947818ef96621896db45b in term 1.
I20240503 14:15:00.792798 8836 raft_consensus.cc:2454] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f156149e6e7947818ef96621896db45b in term 1.
I20240503 14:15:00.793227 8641 leader_election.cc:302] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aab696fc59c4784a0814eadd67ae186, f156149e6e7947818ef96621896db45b; no voters:
I20240503 14:15:00.793509 9332 raft_consensus.cc:2781] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:00.793678 9332 raft_consensus.cc:686] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Becoming Leader. State: Replica: f156149e6e7947818ef96621896db45b, State: Running, Role: LEADER
I20240503 14:15:00.793951 9332 consensus_queue.cc:237] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:00.796629 8424 catalog_manager.cc:5484] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b reported cstate change: term changed from 0 to 1, leader changed from <none> to f156149e6e7947818ef96621896db45b (127.22.242.66). New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:01.143414 9323 consensus_queue.cc:1035] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:01.148597 9359 consensus_queue.cc:1035] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:01.153743 9353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:01.154088 9353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:01.161641 9353 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:15:01.189846 9359 consensus_queue.cc:1035] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240503 14:15:01.196677 9323 consensus_queue.cc:1035] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:01.296789 8836 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:01.297127 8572 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:01.297742 9321 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:01.316917 9321 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:01.393357 9399 mvcc.cc:205] Tried to move back new op lower bound from 7023598392494223360 to 7023598390454165504. Current Snapshot: MvccSnapshot[applied={T|T < 7023598392494223360}]
INSERT report
rows total: 2000
time total: 243.848 ms
time per row: 0.121924 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:15:02.531256 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.69:41037
--local_ip_for_outbound_sockets=127.22.242.69
--tserver_master_addrs=127.22.242.126:36935
--webserver_port=43655
--webserver_interface=127.22.242.69
--builtin_ntp_servers=127.22.242.84:43635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:15:02.862620 9417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:02.862915 9417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:02.863051 9417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:02.870963 9417 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:02.871127 9417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:02.871275 9417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.69
I20240503 14:15:02.880785 9417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:43635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.69:41037
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.242.69
--webserver_port=43655
--tserver_master_addrs=127.22.242.126:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.69
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:02.882112 9417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:02.883857 9417 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:02.900686 9423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:02.900758 9425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:02.900724 9422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:02.901753 9417 server_base.cc:1034] running on GCE node
I20240503 14:15:02.902359 9417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:02.903286 9417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:02.904495 9417 hybrid_clock.cc:648] HybridClock initialized: now 1714745702904443 us; error 40 us; skew 500 ppm
I20240503 14:15:02.904851 9417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:02.914134 9417 webserver.cc:460] Webserver started at http://127.22.242.69:43655/ using document root <none> and password file <none>
I20240503 14:15:02.915138 9417 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:02.915344 9417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:02.922365 9417 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:15:02.925516 9431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:02.927033 9417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:15:02.927207 9417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "85dca8600f3d46b5a5c972c4b343ea09"
format_stamp: "Formatted at 2024-05-03 14:14:53 on dist-test-slave-r3pd"
I20240503 14:15:02.928031 9417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:02.949282 9417 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:02.950147 9417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:02.950433 9417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:02.951459 9417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:02.953094 9417 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:02.953177 9417 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:02.953307 9417 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:02.953364 9417 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:02.984082 9417 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.69:41037
I20240503 14:15:02.984123 9543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.69:41037 every 8 connection(s)
I20240503 14:15:02.985791 9417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:15:02.990836 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 9417
I20240503 14:15:02.991497 23497 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
/tmp/dist-test-taskqfBHnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.242.68:32901
--local_ip_for_outbound_sockets=127.22.242.68
--tserver_master_addrs=127.22.242.126:36935
--webserver_port=42183
--webserver_interface=127.22.242.68
--builtin_ntp_servers=127.22.242.84:43635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:02.999162 9544 heartbeater.cc:347] Connected to a master server at 127.22.242.126:36935
I20240503 14:15:02.999482 9544 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:03.000392 9544 heartbeater.cc:511] Master 127.22.242.126:36935 requested a full tablet report, sending...
I20240503 14:15:03.002195 8433 ts_manager.cc:194] Re-registered known tserver with Master: 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69:41037)
I20240503 14:15:03.003228 8433 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.69:59131
W20240503 14:15:03.342902 9548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:03.343233 9548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:03.343366 9548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:03.351258 9548 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:03.351413 9548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:03.351564 9548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.242.68
I20240503 14:15:03.361074 9548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.242.84:43635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.242.68:32901
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.242.68
--webserver_port=42183
--tserver_master_addrs=127.22.242.126:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.242.68
--log_dir=/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:03.362447 9548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:03.364241 9548 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:03.376924 9555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:03.376832 9554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:03.377352 9557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:03.377779 9548 server_base.cc:1034] running on GCE node
I20240503 14:15:03.378427 9548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:03.379462 9548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:03.380699 9548 hybrid_clock.cc:648] HybridClock initialized: now 1714745703380649 us; error 50 us; skew 500 ppm
I20240503 14:15:03.381095 9548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:03.383591 9548 webserver.cc:460] Webserver started at http://127.22.242.68:42183/ using document root <none> and password file <none>
I20240503 14:15:03.384429 9548 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:03.384565 9548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:03.390578 9548 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:15:03.393704 9563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:03.395323 9548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:15:03.395512 9548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "9920fbe0481e4ee6960a390ac14e9e5c"
format_stamp: "Formatted at 2024-05-03 14:14:52 on dist-test-slave-r3pd"
I20240503 14:15:03.396695 9548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:03.428231 9548 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:03.429113 9548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:03.429397 9548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:03.430449 9548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:03.432022 9548 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:03.432117 9548 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:03.432272 9548 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:03.432324 9548 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:03.468041 9548 rpc_server.cc:307] RPC server started. Bound to: 127.22.242.68:32901
I20240503 14:15:03.468091 9675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.242.68:32901 every 8 connection(s)
I20240503 14:15:03.469516 9548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:15:03.473245 23497 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu as pid 9548
I20240503 14:15:03.484632 9676 heartbeater.cc:347] Connected to a master server at 127.22.242.126:36935
I20240503 14:15:03.484951 9676 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:03.485832 9676 heartbeater.cc:511] Master 127.22.242.126:36935 requested a full tablet report, sending...
I20240503 14:15:03.487550 8433 ts_manager.cc:194] Re-registered known tserver with Master: 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68:32901)
I20240503 14:15:03.488818 8433 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.242.68:51211
I20240503 14:15:03.968784 9479 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:03.970161 9611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:03.995745 8553 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:03.998186 8685 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:04.005992 9544 heartbeater.cc:503] Master 127.22.242.126:36935 was elected leader, sending a full tablet report...
I20240503 14:15:04.010916 8817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:04.178649 8553 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:04.181273 9611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:04.183084 9479 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:04.184003 8817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:04.188521 8685 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:04.383555 9479 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:04.388145 8817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:04.389704 8685 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:04.421027 9611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:04.421361 8553 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:04.491516 9676 heartbeater.cc:503] Master 127.22.242.126:36935 was elected leader, sending a full tablet report...
I20240503 14:15:04.576097 8553 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:04.585613 9479 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:04.586108 9611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:04.597788 8817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:04.604575 8685 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:04.671622 8705 consensus_queue.cc:237] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:04.674729 8572 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:04.675197 8835 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:04.675697 9823 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:04.676124 9824 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:04.679433 9700 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:04.680838 8572 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:04.682358 8835 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
W20240503 14:15:04.683797 8639 consensus_peers.cc:485] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b -> Peer 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69:41037): Couldn't send request to peer 85dca8600f3d46b5a5c972c4b343ea09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 326f8b8a3d754ff68d8f727d18318ff9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:04.686615 8427 catalog_manager.cc:5484] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b reported cstate change: config changed from index -1 to 4, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:04.705211 8572 consensus_queue.cc:237] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:04.708561 8705 raft_consensus.cc:1261] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:04.708703 8835 raft_consensus.cc:1261] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:04.709379 9402 consensus_queue.cc:1035] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:04.709760 9839 consensus_queue.cc:1035] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:04.713122 9839 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:04.715243 8835 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
W20240503 14:15:04.716437 8509 consensus_peers.cc:485] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 -> Peer 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68:32901): Couldn't send request to peer 9920fbe0481e4ee6960a390ac14e9e5c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 029f0fef59a740f88a26ee36fa15907f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:04.716210 8705 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:04.718526 8430 catalog_manager.cc:5484] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: config changed from index -1 to 4, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:04.738299 8835 consensus_queue.cc:237] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:04.740787 8572 raft_consensus.cc:1261] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:04.741469 9849 consensus_queue.cc:1035] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:04.742069 8705 raft_consensus.cc:1261] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:04.742679 9849 consensus_queue.cc:1035] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:04.744501 9176 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:04.745484 8572 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
W20240503 14:15:04.748786 8770 consensus_peers.cc:485] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c -> Peer 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69:41037): Couldn't send request to peer 85dca8600f3d46b5a5c972c4b343ea09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24c71780b3154d54b50cf61ec123d6d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:04.749920 8705 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:04.750418 8430 catalog_manager.cc:5484] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: config changed from index -1 to 4, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "c8d8f771315b46ba84b289cc0189229c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:04.767885 8705 consensus_queue.cc:237] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:04.770242 8835 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:04.770653 8572 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:04.770826 9823 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:04.771325 9700 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:04.773857 9824 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:04.774433 8835 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:04.774930 8572 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
W20240503 14:15:04.778537 8641 consensus_peers.cc:485] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b -> Peer 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68:32901): Couldn't send request to peer 9920fbe0481e4ee6960a390ac14e9e5c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e25a10096a2b4216a0f5714e23b4d643. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:04.779536 8430 catalog_manager.cc:5484] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b reported cstate change: config changed from index -1 to 4, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:04.796032 8705 consensus_queue.cc:237] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:04.798447 8835 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:04.798674 8572 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:04.799063 9823 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:15:04.799397 8639 consensus_peers.cc:485] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b -> Peer 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69:41037): Couldn't send request to peer 85dca8600f3d46b5a5c972c4b343ea09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1af64bab380347dbaa5c38577bb6957c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:04.799443 9700 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:04.802162 9823 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:04.802908 8572 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:04.803761 8835 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:04.807431 8427 catalog_manager.cc:5484] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: config changed from index -1 to 4, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:04.821226 8572 consensus_queue.cc:237] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
W20240503 14:15:04.823647 8509 consensus_peers.cc:485] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 -> Peer 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68:32901): Couldn't send request to peer 9920fbe0481e4ee6960a390ac14e9e5c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b311c3e036d34d3d8947978df14bae01. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:04.823707 8835 raft_consensus.cc:1261] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:04.823796 8705 raft_consensus.cc:1261] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:04.824297 9838 consensus_queue.cc:1035] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:04.824612 9402 consensus_queue.cc:1035] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:04.827315 9839 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:04.828042 8705 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:04.828068 8835 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:04.833346 8430 catalog_manager.cc:5484] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: config changed from index -1 to 4, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:05.154029 9940 ts_tablet_manager.cc:902] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09: Initiating tablet copy from peer f156149e6e7947818ef96621896db45b (127.22.242.66:41441)
I20240503 14:15:05.155694 9940 tablet_copy_client.cc:308] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:41441
I20240503 14:15:05.161639 8726 tablet_copy_service.cc:140] P f156149e6e7947818ef96621896db45b: Received BeginTabletCopySession request for tablet 326f8b8a3d754ff68d8f727d18318ff9 from peer 85dca8600f3d46b5a5c972c4b343ea09 ({username='slave'} at 127.22.242.69:54561)
I20240503 14:15:05.161865 8726 tablet_copy_service.cc:161] P f156149e6e7947818ef96621896db45b: Beginning new tablet copy session on tablet 326f8b8a3d754ff68d8f727d18318ff9 from peer 85dca8600f3d46b5a5c972c4b343ea09 at {username='slave'} at 127.22.242.69:54561: session id = 85dca8600f3d46b5a5c972c4b343ea09-326f8b8a3d754ff68d8f727d18318ff9
I20240503 14:15:05.164449 8726 tablet_copy_source_session.cc:205] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:05.168898 9940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 326f8b8a3d754ff68d8f727d18318ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:05.178965 9940 tablet_copy_client.cc:783] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:05.179301 9940 tablet_copy_client.cc:647] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:05.180456 9946 ts_tablet_manager.cc:902] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c: Initiating tablet copy from peer f156149e6e7947818ef96621896db45b (127.22.242.66:41441)
I20240503 14:15:05.181946 9940 tablet_copy_client.cc:523] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:05.183488 9946 tablet_copy_client.cc:308] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:41441
I20240503 14:15:05.186393 9940 tablet_bootstrap.cc:492] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap starting.
I20240503 14:15:05.190735 8726 tablet_copy_service.cc:140] P f156149e6e7947818ef96621896db45b: Received BeginTabletCopySession request for tablet e25a10096a2b4216a0f5714e23b4d643 from peer 9920fbe0481e4ee6960a390ac14e9e5c ({username='slave'} at 127.22.242.68:57451)
I20240503 14:15:05.190943 8726 tablet_copy_service.cc:161] P f156149e6e7947818ef96621896db45b: Beginning new tablet copy session on tablet e25a10096a2b4216a0f5714e23b4d643 from peer 9920fbe0481e4ee6960a390ac14e9e5c at {username='slave'} at 127.22.242.68:57451: session id = 9920fbe0481e4ee6960a390ac14e9e5c-e25a10096a2b4216a0f5714e23b4d643
I20240503 14:15:05.193125 8726 tablet_copy_source_session.cc:205] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:05.196971 9940 log.cc:824] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:05.197126 9946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25a10096a2b4216a0f5714e23b4d643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:05.207763 9946 tablet_copy_client.cc:783] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:05.208201 9946 tablet_copy_client.cc:647] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:05.210359 9946 tablet_copy_client.cc:523] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:05.213572 9946 tablet_bootstrap.cc:492] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap starting.
I20240503 14:15:05.224102 9946 log.cc:824] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:05.230871 9940 tablet_bootstrap.cc:492] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:05.231336 9940 tablet_bootstrap.cc:492] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap complete.
I20240503 14:15:05.231743 9940 ts_tablet_manager.cc:1367] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09: Time spent bootstrapping tablet: real 0.046s user 0.035s sys 0.008s
I20240503 14:15:05.236301 9940 raft_consensus.cc:348] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:05.236696 9940 raft_consensus.cc:729] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85dca8600f3d46b5a5c972c4b343ea09, State: Initialized, Role: LEARNER
I20240503 14:15:05.237160 9940 consensus_queue.cc:260] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:05.238826 9940 ts_tablet_manager.cc:1398] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20240503 14:15:05.240186 8726 tablet_copy_service.cc:342] P f156149e6e7947818ef96621896db45b: Request end of tablet copy session 85dca8600f3d46b5a5c972c4b343ea09-326f8b8a3d754ff68d8f727d18318ff9 received from {username='slave'} at 127.22.242.69:54561
I20240503 14:15:05.240365 8726 tablet_copy_service.cc:433] P f156149e6e7947818ef96621896db45b: ending tablet copy session 85dca8600f3d46b5a5c972c4b343ea09-326f8b8a3d754ff68d8f727d18318ff9 on tablet 326f8b8a3d754ff68d8f727d18318ff9 with peer 85dca8600f3d46b5a5c972c4b343ea09
I20240503 14:15:05.259851 9946 tablet_bootstrap.cc:492] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:05.260385 9946 tablet_bootstrap.cc:492] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap complete.
I20240503 14:15:05.260802 9946 ts_tablet_manager.cc:1367] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c: Time spent bootstrapping tablet: real 0.047s user 0.039s sys 0.008s
I20240503 14:15:05.263643 9940 ts_tablet_manager.cc:902] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09: Initiating tablet copy from peer f156149e6e7947818ef96621896db45b (127.22.242.66:41441)
I20240503 14:15:05.265234 9940 tablet_copy_client.cc:308] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:41441
I20240503 14:15:05.266029 8726 tablet_copy_service.cc:140] P f156149e6e7947818ef96621896db45b: Received BeginTabletCopySession request for tablet 1af64bab380347dbaa5c38577bb6957c from peer 85dca8600f3d46b5a5c972c4b343ea09 ({username='slave'} at 127.22.242.69:54561)
I20240503 14:15:05.266207 8726 tablet_copy_service.cc:161] P f156149e6e7947818ef96621896db45b: Beginning new tablet copy session on tablet 1af64bab380347dbaa5c38577bb6957c from peer 85dca8600f3d46b5a5c972c4b343ea09 at {username='slave'} at 127.22.242.69:54561: session id = 85dca8600f3d46b5a5c972c4b343ea09-1af64bab380347dbaa5c38577bb6957c
I20240503 14:15:05.266124 9946 raft_consensus.cc:348] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:05.266644 9946 raft_consensus.cc:729] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9920fbe0481e4ee6960a390ac14e9e5c, State: Initialized, Role: LEARNER
I20240503 14:15:05.267246 9946 consensus_queue.cc:260] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:05.269111 9946 ts_tablet_manager.cc:1398] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20240503 14:15:05.269126 8726 tablet_copy_source_session.cc:205] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:05.269922 8726 tablet_copy_service.cc:342] P f156149e6e7947818ef96621896db45b: Request end of tablet copy session 9920fbe0481e4ee6960a390ac14e9e5c-e25a10096a2b4216a0f5714e23b4d643 received from {username='slave'} at 127.22.242.68:57451
I20240503 14:15:05.270067 8726 tablet_copy_service.cc:433] P f156149e6e7947818ef96621896db45b: ending tablet copy session 9920fbe0481e4ee6960a390ac14e9e5c-e25a10096a2b4216a0f5714e23b4d643 on tablet e25a10096a2b4216a0f5714e23b4d643 with peer 9920fbe0481e4ee6960a390ac14e9e5c
I20240503 14:15:05.270385 9940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1af64bab380347dbaa5c38577bb6957c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:05.275275 9940 tablet_copy_client.cc:783] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:05.275503 9940 tablet_copy_client.cc:647] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:05.277015 9940 tablet_copy_client.cc:523] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:05.279314 9940 tablet_bootstrap.cc:492] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap starting.
I20240503 14:15:05.294945 9961 ts_tablet_manager.cc:902] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09: Initiating tablet copy from peer c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:15:05.296586 9961 tablet_copy_client.cc:308] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:37569
I20240503 14:15:05.302909 8857 tablet_copy_service.cc:140] P c8d8f771315b46ba84b289cc0189229c: Received BeginTabletCopySession request for tablet 24c71780b3154d54b50cf61ec123d6d5 from peer 85dca8600f3d46b5a5c972c4b343ea09 ({username='slave'} at 127.22.242.69:57303)
I20240503 14:15:05.303177 8857 tablet_copy_service.cc:161] P c8d8f771315b46ba84b289cc0189229c: Beginning new tablet copy session on tablet 24c71780b3154d54b50cf61ec123d6d5 from peer 85dca8600f3d46b5a5c972c4b343ea09 at {username='slave'} at 127.22.242.69:57303: session id = 85dca8600f3d46b5a5c972c4b343ea09-24c71780b3154d54b50cf61ec123d6d5
I20240503 14:15:05.305603 9946 ts_tablet_manager.cc:902] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c: Initiating tablet copy from peer 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379)
I20240503 14:15:05.306923 8857 tablet_copy_source_session.cc:205] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:05.308883 9961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c71780b3154d54b50cf61ec123d6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:05.312505 9946 tablet_copy_client.cc:308] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:36379
I20240503 14:15:05.315487 9961 tablet_copy_client.cc:783] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:05.315739 9961 tablet_copy_client.cc:647] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:05.317663 9940 tablet_bootstrap.cc:492] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:05.317991 9961 tablet_copy_client.cc:523] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:05.318154 9940 tablet_bootstrap.cc:492] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap complete.
I20240503 14:15:05.318459 9940 ts_tablet_manager.cc:1367] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09: Time spent bootstrapping tablet: real 0.039s user 0.040s sys 0.000s
I20240503 14:15:05.319005 8594 tablet_copy_service.cc:140] P 6aab696fc59c4784a0814eadd67ae186: Received BeginTabletCopySession request for tablet 029f0fef59a740f88a26ee36fa15907f from peer 9920fbe0481e4ee6960a390ac14e9e5c ({username='slave'} at 127.22.242.68:60801)
I20240503 14:15:05.319278 8594 tablet_copy_service.cc:161] P 6aab696fc59c4784a0814eadd67ae186: Beginning new tablet copy session on tablet 029f0fef59a740f88a26ee36fa15907f from peer 9920fbe0481e4ee6960a390ac14e9e5c at {username='slave'} at 127.22.242.68:60801: session id = 9920fbe0481e4ee6960a390ac14e9e5c-029f0fef59a740f88a26ee36fa15907f
I20240503 14:15:05.319588 9940 raft_consensus.cc:348] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:05.319882 9940 raft_consensus.cc:729] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85dca8600f3d46b5a5c972c4b343ea09, State: Initialized, Role: LEARNER
I20240503 14:15:05.320101 9940 consensus_queue.cc:260] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:05.320957 9940 ts_tablet_manager.cc:1398] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:15:05.321024 9961 tablet_bootstrap.cc:492] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap starting.
I20240503 14:15:05.321768 8726 tablet_copy_service.cc:342] P f156149e6e7947818ef96621896db45b: Request end of tablet copy session 85dca8600f3d46b5a5c972c4b343ea09-1af64bab380347dbaa5c38577bb6957c received from {username='slave'} at 127.22.242.69:54561
I20240503 14:15:05.321919 8726 tablet_copy_service.cc:433] P f156149e6e7947818ef96621896db45b: ending tablet copy session 85dca8600f3d46b5a5c972c4b343ea09-1af64bab380347dbaa5c38577bb6957c on tablet 1af64bab380347dbaa5c38577bb6957c with peer 85dca8600f3d46b5a5c972c4b343ea09
I20240503 14:15:05.322742 8594 tablet_copy_source_session.cc:205] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:05.324540 9946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 029f0fef59a740f88a26ee36fa15907f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:05.329694 9946 tablet_copy_client.cc:783] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:05.329926 9946 tablet_copy_client.cc:647] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:05.332047 9946 tablet_copy_client.cc:523] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:05.334576 9946 tablet_bootstrap.cc:492] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap starting.
I20240503 14:15:05.357077 9961 tablet_bootstrap.cc:492] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:05.357537 9961 tablet_bootstrap.cc:492] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap complete.
I20240503 14:15:05.357837 9961 ts_tablet_manager.cc:1367] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09: Time spent bootstrapping tablet: real 0.037s user 0.031s sys 0.004s
I20240503 14:15:05.358650 9961 raft_consensus.cc:348] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:05.358923 9961 raft_consensus.cc:729] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85dca8600f3d46b5a5c972c4b343ea09, State: Initialized, Role: LEARNER
I20240503 14:15:05.359148 9961 consensus_queue.cc:260] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:05.359685 9961 ts_tablet_manager.cc:1398] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:05.360535 8857 tablet_copy_service.cc:342] P c8d8f771315b46ba84b289cc0189229c: Request end of tablet copy session 85dca8600f3d46b5a5c972c4b343ea09-24c71780b3154d54b50cf61ec123d6d5 received from {username='slave'} at 127.22.242.69:57303
I20240503 14:15:05.360723 8857 tablet_copy_service.cc:433] P c8d8f771315b46ba84b289cc0189229c: ending tablet copy session 85dca8600f3d46b5a5c972c4b343ea09-24c71780b3154d54b50cf61ec123d6d5 on tablet 24c71780b3154d54b50cf61ec123d6d5 with peer 85dca8600f3d46b5a5c972c4b343ea09
I20240503 14:15:05.380563 9946 tablet_bootstrap.cc:492] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:05.381057 9946 tablet_bootstrap.cc:492] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap complete.
I20240503 14:15:05.381366 9946 ts_tablet_manager.cc:1367] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c: Time spent bootstrapping tablet: real 0.047s user 0.046s sys 0.000s
I20240503 14:15:05.382488 9946 raft_consensus.cc:348] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:05.382794 9946 raft_consensus.cc:729] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9920fbe0481e4ee6960a390ac14e9e5c, State: Initialized, Role: LEARNER
I20240503 14:15:05.383023 9946 consensus_queue.cc:260] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:05.383575 9946 ts_tablet_manager.cc:1398] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:05.384599 8594 tablet_copy_service.cc:342] P 6aab696fc59c4784a0814eadd67ae186: Request end of tablet copy session 9920fbe0481e4ee6960a390ac14e9e5c-029f0fef59a740f88a26ee36fa15907f received from {username='slave'} at 127.22.242.68:60801
I20240503 14:15:05.384765 8594 tablet_copy_service.cc:433] P 6aab696fc59c4784a0814eadd67ae186: ending tablet copy session 9920fbe0481e4ee6960a390ac14e9e5c-029f0fef59a740f88a26ee36fa15907f on tablet 029f0fef59a740f88a26ee36fa15907f with peer 9920fbe0481e4ee6960a390ac14e9e5c
I20240503 14:15:05.434067 9946 ts_tablet_manager.cc:902] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c: Initiating tablet copy from peer 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379)
I20240503 14:15:05.435209 9946 tablet_copy_client.cc:308] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:36379
I20240503 14:15:05.435895 8594 tablet_copy_service.cc:140] P 6aab696fc59c4784a0814eadd67ae186: Received BeginTabletCopySession request for tablet b311c3e036d34d3d8947978df14bae01 from peer 9920fbe0481e4ee6960a390ac14e9e5c ({username='slave'} at 127.22.242.68:60801)
I20240503 14:15:05.436077 8594 tablet_copy_service.cc:161] P 6aab696fc59c4784a0814eadd67ae186: Beginning new tablet copy session on tablet b311c3e036d34d3d8947978df14bae01 from peer 9920fbe0481e4ee6960a390ac14e9e5c at {username='slave'} at 127.22.242.68:60801: session id = 9920fbe0481e4ee6960a390ac14e9e5c-b311c3e036d34d3d8947978df14bae01
I20240503 14:15:05.438218 8594 tablet_copy_source_session.cc:205] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:05.439313 9946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b311c3e036d34d3d8947978df14bae01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:05.443614 9946 tablet_copy_client.cc:783] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:05.443837 9946 tablet_copy_client.cc:647] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:05.445490 9946 tablet_copy_client.cc:523] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:05.447693 9946 tablet_bootstrap.cc:492] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap starting.
I20240503 14:15:05.481881 9946 tablet_bootstrap.cc:492] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:05.482388 9946 tablet_bootstrap.cc:492] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap complete.
I20240503 14:15:05.482652 9946 ts_tablet_manager.cc:1367] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c: Time spent bootstrapping tablet: real 0.035s user 0.031s sys 0.004s
I20240503 14:15:05.483459 9946 raft_consensus.cc:348] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:05.483695 9946 raft_consensus.cc:729] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9920fbe0481e4ee6960a390ac14e9e5c, State: Initialized, Role: LEARNER
I20240503 14:15:05.483914 9946 consensus_queue.cc:260] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:05.484527 9946 ts_tablet_manager.cc:1398] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:15:05.485595 8594 tablet_copy_service.cc:342] P 6aab696fc59c4784a0814eadd67ae186: Request end of tablet copy session 9920fbe0481e4ee6960a390ac14e9e5c-b311c3e036d34d3d8947978df14bae01 received from {username='slave'} at 127.22.242.68:60801
I20240503 14:15:05.485757 8594 tablet_copy_service.cc:433] P 6aab696fc59c4784a0814eadd67ae186: ending tablet copy session 9920fbe0481e4ee6960a390ac14e9e5c-b311c3e036d34d3d8947978df14bae01 on tablet b311c3e036d34d3d8947978df14bae01 with peer 9920fbe0481e4ee6960a390ac14e9e5c
I20240503 14:15:05.573159 9631 raft_consensus.cc:1203] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:05.590306 9499 raft_consensus.cc:1203] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:05.685074 9499 raft_consensus.cc:1203] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:05.816061 9631 raft_consensus.cc:1203] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:05.836349 9631 raft_consensus.cc:1203] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:05.858525 9499 raft_consensus.cc:1203] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:06.019917 9700 raft_consensus.cc:1050] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b: attempting to promote NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 to VOTER
I20240503 14:15:06.020852 9700 consensus_queue.cc:237] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:06.023252 8572 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:06.023340 9499 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:06.023816 8835 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:06.023876 10007 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:06.024307 9700 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:06.024709 10073 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:06.028715 10007 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.029435 9499 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.032402 8572 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.033344 8427 catalog_manager.cc:5484] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b reported cstate change: config changed from index 4 to 5, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:06.034585 8835 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.041239 8705 consensus_queue.cc:237] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:06.043009 9499 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:06.043505 10073 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:06.043871 8835 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:06.044348 10073 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:06.045701 9700 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.047210 8835 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.048671 8418 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 326f8b8a3d754ff68d8f727d18318ff9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:06.048842 9499 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.051759 8433 catalog_manager.cc:5484] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c reported cstate change: config changed from index 5 to 6, VOTER 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.055752 8553 tablet_service.cc:1514] Processing DeleteTablet for tablet 326f8b8a3d754ff68d8f727d18318ff9 with delete_type TABLET_DATA_TOMBSTONED (TS 6aab696fc59c4784a0814eadd67ae186 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46652
I20240503 14:15:06.056645 10081 tablet_replica.cc:330] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186: stopping tablet replica
I20240503 14:15:06.057037 10081 raft_consensus.cc:2229] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:06.057423 10081 raft_consensus.cc:2258] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:06.058773 10081 ts_tablet_manager.cc:1875] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:06.063347 10081 ts_tablet_manager.cc:1888] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:15:06.063495 10081 log.cc:1196] T 326f8b8a3d754ff68d8f727d18318ff9 P 6aab696fc59c4784a0814eadd67ae186: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/326f8b8a3d754ff68d8f727d18318ff9
I20240503 14:15:06.064503 8419 catalog_manager.cc:4830] TS 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379): tablet 326f8b8a3d754ff68d8f727d18318ff9 (table rebalance_test_table_1 [id=9642b97573e44c3783fd3dddeffb9dfe]) successfully deleted
I20240503 14:15:06.121132 10073 raft_consensus.cc:1050] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b: attempting to promote NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c to VOTER
I20240503 14:15:06.121791 10073 consensus_queue.cc:237] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:06.124141 8835 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:06.124248 8572 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:06.124255 9631 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:06.124773 10073 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:06.125064 9700 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:06.125346 10007 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:06.129449 10007 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.130265 8835 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.133702 9631 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.134351 8572 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.138651 8430 catalog_manager.cc:5484] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c reported cstate change: config changed from index 4 to 5, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.145174 8705 consensus_queue.cc:237] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:06.147115 9630 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:06.148766 10007 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:06.149149 8572 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:06.149708 10007 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:06.151207 9700 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c8d8f771315b46ba84b289cc0189229c (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.154994 8418 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e25a10096a2b4216a0f5714e23b4d643 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:06.155532 9630 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c8d8f771315b46ba84b289cc0189229c (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.156147 8430 catalog_manager.cc:5484] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b reported cstate change: config changed from index 5 to 6, VOTER c8d8f771315b46ba84b289cc0189229c (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:06.159026 8572 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c8d8f771315b46ba84b289cc0189229c (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.160476 8817 tablet_service.cc:1514] Processing DeleteTablet for tablet e25a10096a2b4216a0f5714e23b4d643 with delete_type TABLET_DATA_TOMBSTONED (TS c8d8f771315b46ba84b289cc0189229c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52532
I20240503 14:15:06.161494 10096 tablet_replica.cc:330] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c: stopping tablet replica
I20240503 14:15:06.161823 10096 raft_consensus.cc:2229] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:06.162130 10096 raft_consensus.cc:2258] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:06.163575 10096 ts_tablet_manager.cc:1875] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:06.165158 10038 raft_consensus.cc:1050] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c: attempting to promote NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 to VOTER
I20240503 14:15:06.166146 10038 consensus_queue.cc:237] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:06.174989 10096 ts_tablet_manager.cc:1888] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:15:06.175309 10096 log.cc:1196] T e25a10096a2b4216a0f5714e23b4d643 P c8d8f771315b46ba84b289cc0189229c: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/e25a10096a2b4216a0f5714e23b4d643
I20240503 14:15:06.175689 9499 raft_consensus.cc:1261] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:06.176074 8705 raft_consensus.cc:1261] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:06.176596 8418 catalog_manager.cc:4830] TS c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569): tablet e25a10096a2b4216a0f5714e23b4d643 (table rebalance_test_table_0 [id=1ee49efb58224c89a5d2831f003507e4]) successfully deleted
I20240503 14:15:06.176671 10039 consensus_queue.cc:1035] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:06.177670 10039 consensus_queue.cc:1035] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:06.177867 8572 raft_consensus.cc:1261] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:06.178619 10039 consensus_queue.cc:1035] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:06.194131 9855 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.195392 8572 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.196472 8705 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.199177 9499 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.200547 8433 catalog_manager.cc:5484] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c reported cstate change: config changed from index 4 to 5, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c8d8f771315b46ba84b289cc0189229c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:06.206400 8835 consensus_queue.cc:237] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:06.208786 9499 raft_consensus.cc:1261] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:06.208820 8705 raft_consensus.cc:1261] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:06.209499 10039 consensus_queue.cc:1035] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:06.209885 9855 consensus_queue.cc:1035] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:06.212592 10038 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.213300 9499 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.215548 8418 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 24c71780b3154d54b50cf61ec123d6d5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:06.216442 8705 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.216955 8430 catalog_manager.cc:5484] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c reported cstate change: config changed from index 5 to 6, VOTER 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "c8d8f771315b46ba84b289cc0189229c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:06.221640 8553 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c71780b3154d54b50cf61ec123d6d5 with delete_type TABLET_DATA_TOMBSTONED (TS 6aab696fc59c4784a0814eadd67ae186 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46652
I20240503 14:15:06.222030 10081 tablet_replica.cc:330] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186: stopping tablet replica
I20240503 14:15:06.222262 10081 raft_consensus.cc:2229] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:06.222590 10081 raft_consensus.cc:2258] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:06.223488 10081 ts_tablet_manager.cc:1875] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:06.228469 10081 ts_tablet_manager.cc:1888] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:15:06.228585 10081 log.cc:1196] T 24c71780b3154d54b50cf61ec123d6d5 P 6aab696fc59c4784a0814eadd67ae186: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/24c71780b3154d54b50cf61ec123d6d5
I20240503 14:15:06.229285 8419 catalog_manager.cc:4830] TS 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379): tablet 24c71780b3154d54b50cf61ec123d6d5 (table rebalance_test_table_0 [id=1ee49efb58224c89a5d2831f003507e4]) successfully deleted
I20240503 14:15:06.242637 9700 raft_consensus.cc:1050] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b: attempting to promote NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 to VOTER
I20240503 14:15:06.243561 9700 consensus_queue.cc:237] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:06.246086 9499 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:06.246984 8572 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:06.246991 8835 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:06.247584 9700 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:06.247998 10007 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:06.248360 10073 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:06.251976 10007 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.253935 8572 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.254122 9499 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.256608 8835 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.257252 8433 catalog_manager.cc:5484] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b reported cstate change: config changed from index 4 to 5, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:06.265221 8705 consensus_queue.cc:237] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:06.267290 9499 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:06.267503 8835 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:06.267804 10073 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:06.268513 10073 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:06.270051 9700 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.273341 9499 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.277060 8418 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1af64bab380347dbaa5c38577bb6957c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:06.277885 8427 catalog_manager.cc:5484] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 reported cstate change: config changed from index 5 to 6, VOTER 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65) evicted. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.278457 8835 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:06.282047 8553 tablet_service.cc:1514] Processing DeleteTablet for tablet 1af64bab380347dbaa5c38577bb6957c with delete_type TABLET_DATA_TOMBSTONED (TS 6aab696fc59c4784a0814eadd67ae186 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46652
I20240503 14:15:06.282488 10081 tablet_replica.cc:330] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186: stopping tablet replica
I20240503 14:15:06.282691 10081 raft_consensus.cc:2229] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:06.283056 10081 raft_consensus.cc:2258] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:06.284359 10081 ts_tablet_manager.cc:1875] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:06.290321 10081 ts_tablet_manager.cc:1888] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:15:06.290544 10081 log.cc:1196] T 1af64bab380347dbaa5c38577bb6957c P 6aab696fc59c4784a0814eadd67ae186: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-0/wal/wals/1af64bab380347dbaa5c38577bb6957c
I20240503 14:15:06.291326 8419 catalog_manager.cc:4830] TS 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379): tablet 1af64bab380347dbaa5c38577bb6957c (table rebalance_test_table_2 [id=ca152a2c2e694184b98c458650a677ee]) successfully deleted
I20240503 14:15:06.299507 10041 raft_consensus.cc:1050] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186: attempting to promote NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c to VOTER
I20240503 14:15:06.300472 10041 consensus_queue.cc:237] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:06.303164 8835 raft_consensus.cc:1261] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:06.303763 8705 raft_consensus.cc:1261] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:06.304608 10146 consensus_queue.cc:1035] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:06.305135 9630 raft_consensus.cc:1261] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:06.305171 10041 consensus_queue.cc:1035] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:06.305709 10041 consensus_queue.cc:1035] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:06.323590 10040 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.324394 8835 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.326248 8705 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.330961 8430 catalog_manager.cc:5484] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c reported cstate change: config changed from index 4 to 5, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.332203 9630 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.340368 8572 consensus_queue.cc:237] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:06.343804 9630 raft_consensus.cc:1261] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:06.344041 8705 raft_consensus.cc:1261] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:06.344413 10040 consensus_queue.cc:1035] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:06.344717 10041 consensus_queue.cc:1035] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:06.346716 10146 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c8d8f771315b46ba84b289cc0189229c (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.347276 9630 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c8d8f771315b46ba84b289cc0189229c (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.347333 8705 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c8d8f771315b46ba84b289cc0189229c (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.350695 8419 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 029f0fef59a740f88a26ee36fa15907f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:06.351656 8430 catalog_manager.cc:5484] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c reported cstate change: config changed from index 5 to 6, VOTER c8d8f771315b46ba84b289cc0189229c (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.354992 8817 tablet_service.cc:1514] Processing DeleteTablet for tablet 029f0fef59a740f88a26ee36fa15907f with delete_type TABLET_DATA_TOMBSTONED (TS c8d8f771315b46ba84b289cc0189229c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52532
I20240503 14:15:06.355347 10096 tablet_replica.cc:330] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: stopping tablet replica
I20240503 14:15:06.355561 10096 raft_consensus.cc:2229] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:06.355909 10096 raft_consensus.cc:2258] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:06.356822 10096 ts_tablet_manager.cc:1875] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:06.360949 10096 ts_tablet_manager.cc:1888] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:15:06.361060 10096 log.cc:1196] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/029f0fef59a740f88a26ee36fa15907f
I20240503 14:15:06.361685 8418 catalog_manager.cc:4830] TS c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569): tablet 029f0fef59a740f88a26ee36fa15907f (table rebalance_test_table_2 [id=ca152a2c2e694184b98c458650a677ee]) successfully deleted
I20240503 14:15:06.388072 10146 raft_consensus.cc:1050] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186: attempting to promote NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c to VOTER
I20240503 14:15:06.388799 10146 consensus_queue.cc:237] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:06.391351 8835 raft_consensus.cc:1261] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:06.391352 8705 raft_consensus.cc:1261] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:06.391376 9630 raft_consensus.cc:1261] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:06.391973 10146 consensus_queue.cc:1035] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:06.392350 9838 consensus_queue.cc:1035] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:06.392639 10040 consensus_queue.cc:1035] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:06.395779 9838 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.396409 8705 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.398252 8835 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.401438 8427 catalog_manager.cc:5484] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b reported cstate change: config changed from index 4 to 5, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.401350 9630 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.406828 8574 consensus_queue.cc:237] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:06.408996 9630 raft_consensus.cc:1261] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:06.409567 10146 consensus_queue.cc:1035] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:06.410024 8705 raft_consensus.cc:1261] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:06.411589 10146 consensus_queue.cc:1035] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:06.412314 10041 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c8d8f771315b46ba84b289cc0189229c (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.412947 9631 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c8d8f771315b46ba84b289cc0189229c (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.416496 8419 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b311c3e036d34d3d8947978df14bae01 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:06.418006 8427 catalog_manager.cc:5484] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c reported cstate change: config changed from index 5 to 6, VOTER c8d8f771315b46ba84b289cc0189229c (127.22.242.67) evicted. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.420233 8705 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c8d8f771315b46ba84b289cc0189229c (127.22.242.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:06.421921 8817 tablet_service.cc:1514] Processing DeleteTablet for tablet b311c3e036d34d3d8947978df14bae01 with delete_type TABLET_DATA_TOMBSTONED (TS c8d8f771315b46ba84b289cc0189229c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52532
I20240503 14:15:06.422287 10096 tablet_replica.cc:330] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c: stopping tablet replica
I20240503 14:15:06.422492 10096 raft_consensus.cc:2229] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:06.422838 10096 raft_consensus.cc:2258] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:06.423745 10096 ts_tablet_manager.cc:1875] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:06.428463 10096 ts_tablet_manager.cc:1888] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:15:06.428599 10096 log.cc:1196] T b311c3e036d34d3d8947978df14bae01 P c8d8f771315b46ba84b289cc0189229c: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-2/wal/wals/b311c3e036d34d3d8947978df14bae01
I20240503 14:15:06.429327 8418 catalog_manager.cc:4830] TS c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569): tablet b311c3e036d34d3d8947978df14bae01 (table rebalance_test_table_1 [id=9642b97573e44c3783fd3dddeffb9dfe]) successfully deleted
I20240503 14:15:06.649493 8553 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:06.666136 9479 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:06.667624 8817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:06.671331 9611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:06.681890 8685 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:06.847038 9479 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:06.862758 8685 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:06.863348 8817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:06.863775 8553 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:06.864101 9611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:07.029220 8553 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:07.034781 9611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:07.036339 8817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:07.047425 9479 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:07.055984 8685 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:07.640434 8433 ts_manager.cc:292] Set tserver state for f156149e6e7947818ef96621896db45b to MAINTENANCE_MODE
I20240503 14:15:09.158274 9479 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:09.162075 8553 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:09.173872 9611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:09.186409 8685 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:09.187678 8817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:09.349399 8553 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:09.353294 9479 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:09.363507 8817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:09.366396 8685 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:09.369622 9611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:09.576488 9479 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:09.577149 8553 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:09.586303 9611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:09.588580 8817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:09.598487 8685 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:09.661139 8705 consensus_queue.cc:237] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:09.663228 9499 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:09.663453 8835 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:09.663745 10403 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:09.664036 10233 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:09.666729 10233 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:09.667196 9499 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:09.667460 8835 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:09.672019 8430 catalog_manager.cc:5484] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:09.681196 8705 consensus_queue.cc:237] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:09.684002 8835 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 14:15:09.684427 8641 consensus_peers.cc:485] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b -> Peer 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68:32901): Couldn't send request to peer 9920fbe0481e4ee6960a390ac14e9e5c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 326f8b8a3d754ff68d8f727d18318ff9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:09.684818 10406 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:15:09.686558 9498 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:15:09.687973 10403 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:09.688908 10233 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:15:09.688874 8835 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:09.690598 9498 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:09.692929 8418 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 326f8b8a3d754ff68d8f727d18318ff9 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:09.694427 8433 catalog_manager.cc:5484] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b reported cstate change: config changed from index 7 to 8, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:09.709396 8705 consensus_queue.cc:237] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:09.711097 8835 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:09.711153 9498 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:09.711633 10406 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:09.711879 10233 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:09.714599 10403 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:09.715171 9498 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:09.715212 8835 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:09.720047 8433 catalog_manager.cc:5484] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:09.725425 8705 consensus_queue.cc:237] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
W20240503 14:15:09.729142 8641 consensus_peers.cc:485] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b -> Peer 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68:32901): Couldn't send request to peer 9920fbe0481e4ee6960a390ac14e9e5c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1af64bab380347dbaa5c38577bb6957c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:09.729770 9498 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:15:09.730182 8835 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:15:09.730355 10233 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:15:09.731334 10233 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:15:09.733196 10406 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:09.734354 8835 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:09.736331 8418 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1af64bab380347dbaa5c38577bb6957c with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:09.736491 9499 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:09.740006 8430 catalog_manager.cc:5484] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b reported cstate change: config changed from index 7 to 8, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:09.742022 8705 consensus_queue.cc:237] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:09.744344 9631 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:09.745028 8574 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:09.745625 10406 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:09.745939 10233 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:09.748664 10403 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:09.749411 9631 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:09.753085 8574 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:09.754076 8433 catalog_manager.cc:5484] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:09.760304 8705 consensus_queue.cc:237] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:09.762066 9631 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:15:09.762109 8574 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:15:09.762722 10406 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:15:09.764081 10233 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:15:09.764878 8639 consensus_peers.cc:485] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b -> Peer 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69:41037): Couldn't send request to peer 85dca8600f3d46b5a5c972c4b343ea09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e25a10096a2b4216a0f5714e23b4d643. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:09.765506 10403 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:09.767374 8574 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:09.768994 9631 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:09.769869 8418 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e25a10096a2b4216a0f5714e23b4d643 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:09.772971 8433 catalog_manager.cc:5484] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: config changed from index 7 to 8, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:10.223997 10460 ts_tablet_manager.cc:902] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c: Initiating tablet copy from peer f156149e6e7947818ef96621896db45b (127.22.242.66:41441)
I20240503 14:15:10.225322 10460 tablet_copy_client.cc:308] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:41441
I20240503 14:15:10.226042 8726 tablet_copy_service.cc:140] P f156149e6e7947818ef96621896db45b: Received BeginTabletCopySession request for tablet 326f8b8a3d754ff68d8f727d18318ff9 from peer 9920fbe0481e4ee6960a390ac14e9e5c ({username='slave'} at 127.22.242.68:57451)
I20240503 14:15:10.226272 8726 tablet_copy_service.cc:161] P f156149e6e7947818ef96621896db45b: Beginning new tablet copy session on tablet 326f8b8a3d754ff68d8f727d18318ff9 from peer 9920fbe0481e4ee6960a390ac14e9e5c at {username='slave'} at 127.22.242.68:57451: session id = 9920fbe0481e4ee6960a390ac14e9e5c-326f8b8a3d754ff68d8f727d18318ff9
I20240503 14:15:10.228668 8726 tablet_copy_source_session.cc:205] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:10.229997 10460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 326f8b8a3d754ff68d8f727d18318ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:10.235736 10460 tablet_copy_client.cc:783] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:10.235968 10460 tablet_copy_client.cc:647] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:10.237600 10460 tablet_copy_client.cc:523] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:10.240985 10460 tablet_bootstrap.cc:492] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap starting.
I20240503 14:15:10.276969 10460 tablet_bootstrap.cc:492] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:10.277340 10460 tablet_bootstrap.cc:492] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap complete.
I20240503 14:15:10.277619 10460 ts_tablet_manager.cc:1367] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c: Time spent bootstrapping tablet: real 0.037s user 0.030s sys 0.005s
I20240503 14:15:10.278187 10460 raft_consensus.cc:348] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:10.278396 10460 raft_consensus.cc:729] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9920fbe0481e4ee6960a390ac14e9e5c, State: Initialized, Role: LEARNER
I20240503 14:15:10.278538 10460 consensus_queue.cc:260] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:10.279167 10460 ts_tablet_manager.cc:1398] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:15:10.279990 8726 tablet_copy_service.cc:342] P f156149e6e7947818ef96621896db45b: Request end of tablet copy session 9920fbe0481e4ee6960a390ac14e9e5c-326f8b8a3d754ff68d8f727d18318ff9 received from {username='slave'} at 127.22.242.68:57451
I20240503 14:15:10.280130 8726 tablet_copy_service.cc:433] P f156149e6e7947818ef96621896db45b: ending tablet copy session 9920fbe0481e4ee6960a390ac14e9e5c-326f8b8a3d754ff68d8f727d18318ff9 on tablet 326f8b8a3d754ff68d8f727d18318ff9 with peer 9920fbe0481e4ee6960a390ac14e9e5c
I20240503 14:15:10.307154 10482 ts_tablet_manager.cc:902] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09: Initiating tablet copy from peer f156149e6e7947818ef96621896db45b (127.22.242.66:41441)
I20240503 14:15:10.308387 10482 tablet_copy_client.cc:308] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:41441
I20240503 14:15:10.309062 8726 tablet_copy_service.cc:140] P f156149e6e7947818ef96621896db45b: Received BeginTabletCopySession request for tablet e25a10096a2b4216a0f5714e23b4d643 from peer 85dca8600f3d46b5a5c972c4b343ea09 ({username='slave'} at 127.22.242.69:54561)
I20240503 14:15:10.309211 8726 tablet_copy_service.cc:161] P f156149e6e7947818ef96621896db45b: Beginning new tablet copy session on tablet e25a10096a2b4216a0f5714e23b4d643 from peer 85dca8600f3d46b5a5c972c4b343ea09 at {username='slave'} at 127.22.242.69:54561: session id = 85dca8600f3d46b5a5c972c4b343ea09-e25a10096a2b4216a0f5714e23b4d643
I20240503 14:15:10.311057 8726 tablet_copy_source_session.cc:205] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:10.312083 10482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25a10096a2b4216a0f5714e23b4d643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:10.316291 10482 tablet_copy_client.cc:783] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:10.316524 10482 tablet_copy_client.cc:647] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:10.318110 10482 tablet_copy_client.cc:523] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:10.320412 10482 tablet_bootstrap.cc:492] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap starting.
I20240503 14:15:10.333266 10460 ts_tablet_manager.cc:902] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c: Initiating tablet copy from peer f156149e6e7947818ef96621896db45b (127.22.242.66:41441)
I20240503 14:15:10.334621 10460 tablet_copy_client.cc:308] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:41441
I20240503 14:15:10.335319 8726 tablet_copy_service.cc:140] P f156149e6e7947818ef96621896db45b: Received BeginTabletCopySession request for tablet 1af64bab380347dbaa5c38577bb6957c from peer 9920fbe0481e4ee6960a390ac14e9e5c ({username='slave'} at 127.22.242.68:57451)
I20240503 14:15:10.335477 8726 tablet_copy_service.cc:161] P f156149e6e7947818ef96621896db45b: Beginning new tablet copy session on tablet 1af64bab380347dbaa5c38577bb6957c from peer 9920fbe0481e4ee6960a390ac14e9e5c at {username='slave'} at 127.22.242.68:57451: session id = 9920fbe0481e4ee6960a390ac14e9e5c-1af64bab380347dbaa5c38577bb6957c
I20240503 14:15:10.337389 8726 tablet_copy_source_session.cc:205] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:10.338472 10460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1af64bab380347dbaa5c38577bb6957c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:10.342797 10460 tablet_copy_client.cc:783] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:10.343084 10460 tablet_copy_client.cc:647] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:10.344628 10460 tablet_copy_client.cc:523] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:10.347211 10460 tablet_bootstrap.cc:492] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap starting.
I20240503 14:15:10.354470 10482 tablet_bootstrap.cc:492] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:10.354820 10482 tablet_bootstrap.cc:492] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap complete.
I20240503 14:15:10.355043 10482 ts_tablet_manager.cc:1367] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09: Time spent bootstrapping tablet: real 0.035s user 0.032s sys 0.000s
I20240503 14:15:10.355685 10482 raft_consensus.cc:348] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:10.355895 10482 raft_consensus.cc:729] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85dca8600f3d46b5a5c972c4b343ea09, State: Initialized, Role: LEARNER
I20240503 14:15:10.356074 10482 consensus_queue.cc:260] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:10.356839 10482 ts_tablet_manager.cc:1398] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:15:10.357712 8726 tablet_copy_service.cc:342] P f156149e6e7947818ef96621896db45b: Request end of tablet copy session 85dca8600f3d46b5a5c972c4b343ea09-e25a10096a2b4216a0f5714e23b4d643 received from {username='slave'} at 127.22.242.69:54561
I20240503 14:15:10.357838 8726 tablet_copy_service.cc:433] P f156149e6e7947818ef96621896db45b: ending tablet copy session 85dca8600f3d46b5a5c972c4b343ea09-e25a10096a2b4216a0f5714e23b4d643 on tablet e25a10096a2b4216a0f5714e23b4d643 with peer 85dca8600f3d46b5a5c972c4b343ea09
I20240503 14:15:10.378036 10460 tablet_bootstrap.cc:492] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:10.378479 10460 tablet_bootstrap.cc:492] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap complete.
I20240503 14:15:10.378726 10460 ts_tablet_manager.cc:1367] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.004s
I20240503 14:15:10.379388 10460 raft_consensus.cc:348] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:10.379593 10460 raft_consensus.cc:729] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9920fbe0481e4ee6960a390ac14e9e5c, State: Initialized, Role: LEARNER
I20240503 14:15:10.379805 10460 consensus_queue.cc:260] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:10.380287 10460 ts_tablet_manager.cc:1398] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 14:15:10.381161 8726 tablet_copy_service.cc:342] P f156149e6e7947818ef96621896db45b: Request end of tablet copy session 9920fbe0481e4ee6960a390ac14e9e5c-1af64bab380347dbaa5c38577bb6957c received from {username='slave'} at 127.22.242.68:57451
I20240503 14:15:10.381280 8726 tablet_copy_service.cc:433] P f156149e6e7947818ef96621896db45b: ending tablet copy session 9920fbe0481e4ee6960a390ac14e9e5c-1af64bab380347dbaa5c38577bb6957c on tablet 1af64bab380347dbaa5c38577bb6957c with peer 9920fbe0481e4ee6960a390ac14e9e5c
I20240503 14:15:10.791217 9631 raft_consensus.cc:1203] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:15:10.907902 9631 raft_consensus.cc:1203] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:15:10.930941 9499 raft_consensus.cc:1203] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:15:11.391286 10233 raft_consensus.cc:1050] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b: attempting to promote NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 to VOTER
I20240503 14:15:11.392128 10233 consensus_queue.cc:237] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:11.394856 9631 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:15:11.395529 10524 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:15:11.395808 8574 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:15:11.396700 9499 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:15:11.397292 10525 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:15:11.397974 10525 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:15:11.407277 10525 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:11.411042 9499 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:11.411998 10524 raft_consensus.cc:1050] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b: attempting to promote NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c to VOTER
I20240503 14:15:11.412875 10524 consensus_queue.cc:237] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:11.415701 9630 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:11.416456 8427 catalog_manager.cc:5484] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09 reported cstate change: config changed from index 8 to 9, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:11.424551 9629 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:15:11.429235 10233 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:15:11.430668 8574 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:11.456110 9499 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:15:11.456465 8835 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:15:11.456944 10524 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:15:11.457370 10564 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:15:11.474323 10564 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:11.479744 9630 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:11.482585 8835 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:11.486176 8426 catalog_manager.cc:5484] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b reported cstate change: config changed from index 8 to 9, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:11.490653 9499 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:11.506677 8706 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "326f8b8a3d754ff68d8f727d18318ff9"
dest_uuid: "f156149e6e7947818ef96621896db45b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35742
I20240503 14:15:11.506909 8706 raft_consensus.cc:595] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Received request to transfer leadership
I20240503 14:15:11.509876 10524 raft_consensus.cc:1050] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b: attempting to promote NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c to VOTER
I20240503 14:15:11.510705 10524 consensus_queue.cc:237] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:11.512850 8835 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:15:11.513388 10575 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:15:11.513515 9630 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:15:11.513507 9499 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:15:11.514086 10575 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:15:11.514647 10564 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:15:11.518337 10524 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:11.518873 9499 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:11.518970 9630 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:11.524674 8426 catalog_manager.cc:5484] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c reported cstate change: config changed from index 8 to 9, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:11.528240 8835 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:11.540488 8706 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1af64bab380347dbaa5c38577bb6957c"
dest_uuid: "f156149e6e7947818ef96621896db45b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35760
I20240503 14:15:11.540673 8706 raft_consensus.cc:595] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Received request to transfer leadership
I20240503 14:15:11.758044 8706 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e25a10096a2b4216a0f5714e23b4d643"
dest_uuid: "f156149e6e7947818ef96621896db45b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35776
I20240503 14:15:11.758242 8706 raft_consensus.cc:595] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Received request to transfer leadership
I20240503 14:15:11.774116 8706 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "326f8b8a3d754ff68d8f727d18318ff9"
dest_uuid: "f156149e6e7947818ef96621896db45b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35788
I20240503 14:15:11.774308 8706 raft_consensus.cc:595] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Received request to transfer leadership
I20240503 14:15:11.789921 8706 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1af64bab380347dbaa5c38577bb6957c"
dest_uuid: "f156149e6e7947818ef96621896db45b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35806
I20240503 14:15:11.790115 8706 raft_consensus.cc:595] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Received request to transfer leadership
I20240503 14:15:11.804373 10525 raft_consensus.cc:979] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b: : Instructing follower 6aab696fc59c4784a0814eadd67ae186 to start an election
I20240503 14:15:11.804549 10524 raft_consensus.cc:1067] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Signalling peer 6aab696fc59c4784a0814eadd67ae186 to start an election
I20240503 14:15:11.805414 8574 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e25a10096a2b4216a0f5714e23b4d643"
dest_uuid: "6aab696fc59c4784a0814eadd67ae186"
from {username='slave'} at 127.22.242.66:54465
I20240503 14:15:11.805649 8574 raft_consensus.cc:482] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:15:11.805781 8574 raft_consensus.cc:3037] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:11.807793 8574 raft_consensus.cc:504] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:11.809178 8706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25a10096a2b4216a0f5714e23b4d643" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "f156149e6e7947818ef96621896db45b"
I20240503 14:15:11.808805 8574 leader_election.cc:288] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 2 election: Requested vote from peers f156149e6e7947818ef96621896db45b (127.22.242.66:41441), 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68:32901), 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69:41037)
I20240503 14:15:11.809408 8706 raft_consensus.cc:3032] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:15:11.809509 8706 raft_consensus.cc:729] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f156149e6e7947818ef96621896db45b, State: Running, Role: LEADER
I20240503 14:15:11.809514 9630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25a10096a2b4216a0f5714e23b4d643" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "9920fbe0481e4ee6960a390ac14e9e5c"
I20240503 14:15:11.809836 9630 raft_consensus.cc:3037] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:11.809870 8706 consensus_queue.cc:260] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:11.810369 8706 raft_consensus.cc:3037] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:11.812916 9630 raft_consensus.cc:2454] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 2.
I20240503 14:15:11.813129 8706 raft_consensus.cc:2454] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 2.
I20240503 14:15:11.813592 8508 leader_election.cc:302] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6aab696fc59c4784a0814eadd67ae186, 9920fbe0481e4ee6960a390ac14e9e5c, f156149e6e7947818ef96621896db45b; no voters:
I20240503 14:15:11.813910 10583 raft_consensus.cc:2781] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:11.814034 10583 raft_consensus.cc:686] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 2 LEADER]: Becoming Leader. State: Replica: 6aab696fc59c4784a0814eadd67ae186, State: Running, Role: LEADER
I20240503 14:15:11.814327 10583 consensus_queue.cc:237] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:11.817224 8430 catalog_manager.cc:5484] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: term changed from 1 to 2, leader changed from f156149e6e7947818ef96621896db45b (127.22.242.66) to 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65). New cstate: current_term: 2 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:11.818321 9499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25a10096a2b4216a0f5714e23b4d643" candidate_uuid: "6aab696fc59c4784a0814eadd67ae186" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "85dca8600f3d46b5a5c972c4b343ea09"
I20240503 14:15:11.818603 9499 raft_consensus.cc:3037] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:11.820766 9499 raft_consensus.cc:2454] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aab696fc59c4784a0814eadd67ae186 in term 2.
I20240503 14:15:11.873286 10525 raft_consensus.cc:979] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b: : Instructing follower c8d8f771315b46ba84b289cc0189229c to start an election
I20240503 14:15:11.873454 10525 raft_consensus.cc:1067] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Signalling peer c8d8f771315b46ba84b289cc0189229c to start an election
I20240503 14:15:11.874127 8835 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "326f8b8a3d754ff68d8f727d18318ff9"
dest_uuid: "c8d8f771315b46ba84b289cc0189229c"
from {username='slave'} at 127.22.242.66:46541
I20240503 14:15:11.874338 8835 raft_consensus.cc:482] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:15:11.874466 8835 raft_consensus.cc:3037] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:11.876400 8835 raft_consensus.cc:504] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:11.877669 9499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "326f8b8a3d754ff68d8f727d18318ff9" candidate_uuid: "c8d8f771315b46ba84b289cc0189229c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "85dca8600f3d46b5a5c972c4b343ea09"
I20240503 14:15:11.877965 9499 raft_consensus.cc:3037] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:11.877969 8835 leader_election.cc:288] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [CANDIDATE]: Term 2 election: Requested vote from peers f156149e6e7947818ef96621896db45b (127.22.242.66:41441), 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69:41037), 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68:32901)
I20240503 14:15:11.878572 8706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "326f8b8a3d754ff68d8f727d18318ff9" candidate_uuid: "c8d8f771315b46ba84b289cc0189229c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "f156149e6e7947818ef96621896db45b"
I20240503 14:15:11.878754 8706 raft_consensus.cc:3032] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:15:11.878902 8706 raft_consensus.cc:729] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f156149e6e7947818ef96621896db45b, State: Running, Role: LEADER
I20240503 14:15:11.879273 8706 consensus_queue.cc:260] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:11.879814 8706 raft_consensus.cc:3037] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:11.880923 9499 raft_consensus.cc:2454] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d8f771315b46ba84b289cc0189229c in term 2.
I20240503 14:15:11.882185 8706 raft_consensus.cc:2454] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d8f771315b46ba84b289cc0189229c in term 2.
I20240503 14:15:11.882589 8771 leader_election.cc:302] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 85dca8600f3d46b5a5c972c4b343ea09, c8d8f771315b46ba84b289cc0189229c, f156149e6e7947818ef96621896db45b; no voters:
I20240503 14:15:11.882922 10038 raft_consensus.cc:2781] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:11.883411 10038 raft_consensus.cc:686] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 2 LEADER]: Becoming Leader. State: Replica: c8d8f771315b46ba84b289cc0189229c, State: Running, Role: LEADER
I20240503 14:15:11.883738 10038 consensus_queue.cc:237] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:11.885082 9630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "326f8b8a3d754ff68d8f727d18318ff9" candidate_uuid: "c8d8f771315b46ba84b289cc0189229c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "9920fbe0481e4ee6960a390ac14e9e5c"
I20240503 14:15:11.885272 9630 raft_consensus.cc:3037] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:11.886406 8430 catalog_manager.cc:5484] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c reported cstate change: term changed from 1 to 2, leader changed from f156149e6e7947818ef96621896db45b (127.22.242.66) to c8d8f771315b46ba84b289cc0189229c (127.22.242.67). New cstate: current_term: 2 leader_uuid: "c8d8f771315b46ba84b289cc0189229c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:11.888070 9630 raft_consensus.cc:2454] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d8f771315b46ba84b289cc0189229c in term 2.
I20240503 14:15:11.901248 10524 raft_consensus.cc:979] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b: : Instructing follower 9920fbe0481e4ee6960a390ac14e9e5c to start an election
I20240503 14:15:11.901408 10524 raft_consensus.cc:1067] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Signalling peer 9920fbe0481e4ee6960a390ac14e9e5c to start an election
I20240503 14:15:11.901999 9630 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1af64bab380347dbaa5c38577bb6957c"
dest_uuid: "9920fbe0481e4ee6960a390ac14e9e5c"
from {username='slave'} at 127.22.242.66:49253
I20240503 14:15:11.902258 9630 raft_consensus.cc:482] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:15:11.902364 9630 raft_consensus.cc:3037] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:11.904639 9630 raft_consensus.cc:504] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:11.906184 9630 leader_election.cc:288] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [CANDIDATE]: Term 2 election: Requested vote from peers f156149e6e7947818ef96621896db45b (127.22.242.66:41441), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569), 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69:41037)
I20240503 14:15:11.906581 8706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af64bab380347dbaa5c38577bb6957c" candidate_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "f156149e6e7947818ef96621896db45b"
I20240503 14:15:11.906791 8706 raft_consensus.cc:3032] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:15:11.906905 8706 raft_consensus.cc:729] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f156149e6e7947818ef96621896db45b, State: Running, Role: LEADER
I20240503 14:15:11.907231 8706 consensus_queue.cc:260] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:11.907831 8706 raft_consensus.cc:3037] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:11.911592 8706 raft_consensus.cc:2454] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9920fbe0481e4ee6960a390ac14e9e5c in term 2.
I20240503 14:15:11.923115 8835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af64bab380347dbaa5c38577bb6957c" candidate_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "c8d8f771315b46ba84b289cc0189229c"
I20240503 14:15:11.923348 8835 raft_consensus.cc:3037] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:11.923452 9499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af64bab380347dbaa5c38577bb6957c" candidate_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "85dca8600f3d46b5a5c972c4b343ea09"
I20240503 14:15:11.923671 9499 raft_consensus.cc:3037] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:11.925689 9499 raft_consensus.cc:2454] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9920fbe0481e4ee6960a390ac14e9e5c in term 2.
I20240503 14:15:11.926118 8835 raft_consensus.cc:2454] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9920fbe0481e4ee6960a390ac14e9e5c in term 2.
I20240503 14:15:11.926122 9564 leader_election.cc:302] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 85dca8600f3d46b5a5c972c4b343ea09, 9920fbe0481e4ee6960a390ac14e9e5c, f156149e6e7947818ef96621896db45b; no voters:
I20240503 14:15:11.926573 10571 raft_consensus.cc:2781] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:11.927101 10571 raft_consensus.cc:686] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [term 2 LEADER]: Becoming Leader. State: Replica: 9920fbe0481e4ee6960a390ac14e9e5c, State: Running, Role: LEADER
I20240503 14:15:11.927654 10571 consensus_queue.cc:237] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:11.931423 8430 catalog_manager.cc:5484] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c reported cstate change: term changed from 1 to 2, leader changed from f156149e6e7947818ef96621896db45b (127.22.242.66) to 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68). New cstate: current_term: 2 leader_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:12.203956 9499 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09 [term 2 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:15:12.204535 10583 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:15:12.260727 8706 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 2 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:15:12.261286 10583 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:15:12.266587 9630 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [term 2 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:15:12.267364 10676 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:15:12.286296 9499 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [term 2 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:15:12.286855 10038 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:15:12.379531 9499 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [term 2 FOLLOWER]: Refusing update from remote peer 9920fbe0481e4ee6960a390ac14e9e5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:15:12.380206 10571 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:15:12.421355 9630 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c [term 2 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:15:12.422070 10038 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:15:12.428427 8706 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 2 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:15:12.429268 10038 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:15:12.437561 8835 consensus_queue.cc:237] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:12.439142 9630 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c [term 2 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:15:12.439707 10679 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:15:12.440305 9499 raft_consensus.cc:1261] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [term 2 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:15:12.440832 10679 consensus_queue.cc:1035] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:15:12.442606 9630 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P 9920fbe0481e4ee6960a390ac14e9e5c [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:12.441967 10038 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P c8d8f771315b46ba84b289cc0189229c [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:12.444738 9499 raft_consensus.cc:2932] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:12.447109 8418 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 326f8b8a3d754ff68d8f727d18318ff9 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:12.448742 8426 catalog_manager.cc:5484] T 326f8b8a3d754ff68d8f727d18318ff9 P 85dca8600f3d46b5a5c972c4b343ea09 reported cstate change: config changed from index 9 to 11, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New cstate: current_term: 2 leader_uuid: "c8d8f771315b46ba84b289cc0189229c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:12.451494 8706 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 2 FOLLOWER]: Refusing update from remote peer 9920fbe0481e4ee6960a390ac14e9e5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:15:12.452229 10571 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:15:12.454806 8685 tablet_service.cc:1514] Processing DeleteTablet for tablet 326f8b8a3d754ff68d8f727d18318ff9 with delete_type TABLET_DATA_TOMBSTONED (TS f156149e6e7947818ef96621896db45b not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:55852
I20240503 14:15:12.459542 8835 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 2 FOLLOWER]: Refusing update from remote peer 9920fbe0481e4ee6960a390ac14e9e5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:15:12.461292 10571 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:15:12.462438 10687 tablet_replica.cc:330] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b: stopping tablet replica
I20240503 14:15:12.463121 10687 raft_consensus.cc:2229] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:12.463428 10687 raft_consensus.cc:2258] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:12.466506 10687 ts_tablet_manager.cc:1875] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:12.473315 10687 ts_tablet_manager.cc:1888] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 14:15:12.473562 10687 log.cc:1196] T 326f8b8a3d754ff68d8f727d18318ff9 P f156149e6e7947818ef96621896db45b: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/326f8b8a3d754ff68d8f727d18318ff9
I20240503 14:15:12.474622 8418 catalog_manager.cc:4830] TS f156149e6e7947818ef96621896db45b (127.22.242.66:41441): tablet 326f8b8a3d754ff68d8f727d18318ff9 (table rebalance_test_table_1 [id=9642b97573e44c3783fd3dddeffb9dfe]) successfully deleted
I20240503 14:15:12.717970 8574 consensus_queue.cc:237] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:12.720088 8706 raft_consensus.cc:1261] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:12.720078 9630 raft_consensus.cc:1261] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:12.720656 10676 consensus_queue.cc:1035] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:12.720921 10583 consensus_queue.cc:1035] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:12.723436 10459 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:12.724130 9630 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:12.724283 8706 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:12.728547 8426 catalog_manager.cc:5484] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:12.734062 8574 consensus_queue.cc:237] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:12.736554 9630 raft_consensus.cc:1261] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:15:12.737447 10583 consensus_queue.cc:1035] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:15:12.737828 8706 raft_consensus.cc:1261] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:15:12.738328 10583 consensus_queue.cc:1035] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:15:12.739153 8507 consensus_peers.cc:485] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 -> Peer 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69:41037): Couldn't send request to peer 85dca8600f3d46b5a5c972c4b343ea09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b311c3e036d34d3d8947978df14bae01. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:12.740120 10676 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:12.740897 9630 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:12.744066 8419 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b311c3e036d34d3d8947978df14bae01 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:12.745062 8430 catalog_manager.cc:5484] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: config changed from index 7 to 8, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:12.746232 8705 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:12.756727 8574 consensus_queue.cc:237] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:12.758400 9630 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [term 2 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:15:12.758517 9499 raft_consensus.cc:1261] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09 [term 2 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:15:12.758978 10583 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:15:12.759472 10583 consensus_queue.cc:1035] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:15:12.761803 10676 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:12.764837 9630 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P 9920fbe0481e4ee6960a390ac14e9e5c [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:12.765003 9499 raft_consensus.cc:2932] T e25a10096a2b4216a0f5714e23b4d643 P 85dca8600f3d46b5a5c972c4b343ea09 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:12.766701 8430 catalog_manager.cc:5484] T e25a10096a2b4216a0f5714e23b4d643 P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: config changed from index 9 to 11, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New cstate: current_term: 2 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:12.769908 8419 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e25a10096a2b4216a0f5714e23b4d643 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:12.772341 8685 tablet_service.cc:1514] Processing DeleteTablet for tablet e25a10096a2b4216a0f5714e23b4d643 with delete_type TABLET_DATA_TOMBSTONED (TS f156149e6e7947818ef96621896db45b not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:55852
I20240503 14:15:12.772722 10687 tablet_replica.cc:330] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b: stopping tablet replica
I20240503 14:15:12.772941 10687 raft_consensus.cc:2229] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:12.773173 10687 raft_consensus.cc:2258] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:12.774024 10687 ts_tablet_manager.cc:1875] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:12.778160 10687 ts_tablet_manager.cc:1888] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 14:15:12.778287 10687 log.cc:1196] T e25a10096a2b4216a0f5714e23b4d643 P f156149e6e7947818ef96621896db45b: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/e25a10096a2b4216a0f5714e23b4d643
I20240503 14:15:12.779122 8418 catalog_manager.cc:4830] TS f156149e6e7947818ef96621896db45b (127.22.242.66:41441): tablet e25a10096a2b4216a0f5714e23b4d643 (table rebalance_test_table_0 [id=1ee49efb58224c89a5d2831f003507e4]) successfully deleted
I20240503 14:15:12.872293 9630 consensus_queue.cc:237] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:12.874028 8835 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 2 FOLLOWER]: Refusing update from remote peer 9920fbe0481e4ee6960a390ac14e9e5c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:15:12.874043 9499 raft_consensus.cc:1261] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [term 2 FOLLOWER]: Refusing update from remote peer 9920fbe0481e4ee6960a390ac14e9e5c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:15:12.874543 10688 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:15:12.874863 10571 consensus_queue.cc:1035] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:15:12.877029 10686 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P 9920fbe0481e4ee6960a390ac14e9e5c [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:12.877579 9499 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:12.880800 8419 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1af64bab380347dbaa5c38577bb6957c with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:12.881114 8835 raft_consensus.cc:2932] T 1af64bab380347dbaa5c38577bb6957c P c8d8f771315b46ba84b289cc0189229c [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:12.881520 8426 catalog_manager.cc:5484] T 1af64bab380347dbaa5c38577bb6957c P 85dca8600f3d46b5a5c972c4b343ea09 reported cstate change: config changed from index 9 to 11, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New cstate: current_term: 2 leader_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:12.888566 8685 tablet_service.cc:1514] Processing DeleteTablet for tablet 1af64bab380347dbaa5c38577bb6957c with delete_type TABLET_DATA_TOMBSTONED (TS f156149e6e7947818ef96621896db45b not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:55852
I20240503 14:15:12.888919 10687 tablet_replica.cc:330] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b: stopping tablet replica
I20240503 14:15:12.889107 10687 raft_consensus.cc:2229] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:12.889349 10687 raft_consensus.cc:2258] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:12.890087 10687 ts_tablet_manager.cc:1875] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:12.894286 10687 ts_tablet_manager.cc:1888] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 14:15:12.894384 10687 log.cc:1196] T 1af64bab380347dbaa5c38577bb6957c P f156149e6e7947818ef96621896db45b: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/1af64bab380347dbaa5c38577bb6957c
I20240503 14:15:12.895004 8418 catalog_manager.cc:4830] TS f156149e6e7947818ef96621896db45b (127.22.242.66:41441): tablet 1af64bab380347dbaa5c38577bb6957c (table rebalance_test_table_2 [id=ca152a2c2e694184b98c458650a677ee]) successfully deleted
I20240503 14:15:13.007033 8705 consensus_queue.cc:237] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:13.009521 8572 raft_consensus.cc:1261] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:13.009546 8835 raft_consensus.cc:1261] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:13.010138 10735 consensus_queue.cc:1035] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:13.010453 10681 consensus_queue.cc:1035] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:13.013319 10681 raft_consensus.cc:2932] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } }
I20240503 14:15:13.013763 8572 raft_consensus.cc:2932] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } }
I20240503 14:15:13.013947 8835 raft_consensus.cc:2932] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } }
I20240503 14:15:13.018051 8430 catalog_manager.cc:5484] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } }
I20240503 14:15:13.023670 8705 consensus_queue.cc:237] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:13.025903 8835 raft_consensus.cc:1261] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:13.026346 8572 raft_consensus.cc:1261] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:15:13.026825 8639 consensus_peers.cc:485] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b -> Peer 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69:41037): Couldn't send request to peer 85dca8600f3d46b5a5c972c4b343ea09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a061e8fe907c4f5899d1d90cf1c06754. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:13.027328 10681 consensus_queue.cc:1035] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:13.027604 10735 consensus_queue.cc:1035] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:13.030508 10735 raft_consensus.cc:2932] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:13.031265 8835 raft_consensus.cc:2932] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:13.033004 8572 raft_consensus.cc:2932] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:13.034137 8418 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a061e8fe907c4f5899d1d90cf1c06754 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:13.035784 8426 catalog_manager.cc:5484] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c reported cstate change: config changed from index 4 to 5, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:13.153271 10754 ts_tablet_manager.cc:902] T b311c3e036d34d3d8947978df14bae01 P 85dca8600f3d46b5a5c972c4b343ea09: Initiating tablet copy from peer 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379)
I20240503 14:15:13.154826 10754 tablet_copy_client.cc:308] T b311c3e036d34d3d8947978df14bae01 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:36379
I20240503 14:15:13.160583 8594 tablet_copy_service.cc:140] P 6aab696fc59c4784a0814eadd67ae186: Received BeginTabletCopySession request for tablet b311c3e036d34d3d8947978df14bae01 from peer 85dca8600f3d46b5a5c972c4b343ea09 ({username='slave'} at 127.22.242.69:52357)
I20240503 14:15:13.160712 8594 tablet_copy_service.cc:161] P 6aab696fc59c4784a0814eadd67ae186: Beginning new tablet copy session on tablet b311c3e036d34d3d8947978df14bae01 from peer 85dca8600f3d46b5a5c972c4b343ea09 at {username='slave'} at 127.22.242.69:52357: session id = 85dca8600f3d46b5a5c972c4b343ea09-b311c3e036d34d3d8947978df14bae01
I20240503 14:15:13.162523 8594 tablet_copy_source_session.cc:205] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:13.163686 10754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b311c3e036d34d3d8947978df14bae01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:13.167784 10754 tablet_copy_client.cc:783] T b311c3e036d34d3d8947978df14bae01 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:13.168046 10754 tablet_copy_client.cc:647] T b311c3e036d34d3d8947978df14bae01 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:13.169610 10754 tablet_copy_client.cc:523] T b311c3e036d34d3d8947978df14bae01 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:13.171815 10754 tablet_bootstrap.cc:492] T b311c3e036d34d3d8947978df14bae01 P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap starting.
I20240503 14:15:13.202028 10754 tablet_bootstrap.cc:492] T b311c3e036d34d3d8947978df14bae01 P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:13.202495 10754 tablet_bootstrap.cc:492] T b311c3e036d34d3d8947978df14bae01 P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap complete.
I20240503 14:15:13.202775 10754 ts_tablet_manager.cc:1367] T b311c3e036d34d3d8947978df14bae01 P 85dca8600f3d46b5a5c972c4b343ea09: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.004s
I20240503 14:15:13.203418 10754 raft_consensus.cc:348] T b311c3e036d34d3d8947978df14bae01 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:13.203650 10754 raft_consensus.cc:729] T b311c3e036d34d3d8947978df14bae01 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85dca8600f3d46b5a5c972c4b343ea09, State: Initialized, Role: LEARNER
I20240503 14:15:13.203840 10754 consensus_queue.cc:260] T b311c3e036d34d3d8947978df14bae01 P 85dca8600f3d46b5a5c972c4b343ea09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:13.204339 10754 ts_tablet_manager.cc:1398] T b311c3e036d34d3d8947978df14bae01 P 85dca8600f3d46b5a5c972c4b343ea09: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 14:15:13.205212 8594 tablet_copy_service.cc:342] P 6aab696fc59c4784a0814eadd67ae186: Request end of tablet copy session 85dca8600f3d46b5a5c972c4b343ea09-b311c3e036d34d3d8947978df14bae01 received from {username='slave'} at 127.22.242.69:52357
I20240503 14:15:13.205332 8594 tablet_copy_service.cc:433] P 6aab696fc59c4784a0814eadd67ae186: ending tablet copy session 85dca8600f3d46b5a5c972c4b343ea09-b311c3e036d34d3d8947978df14bae01 on tablet b311c3e036d34d3d8947978df14bae01 with peer 85dca8600f3d46b5a5c972c4b343ea09
I20240503 14:15:13.267076 8835 consensus_queue.cc:237] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:13.269291 9499 raft_consensus.cc:1261] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:13.269466 8705 raft_consensus.cc:1261] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:13.269845 10679 consensus_queue.cc:1035] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:13.270128 10768 consensus_queue.cc:1035] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:13.273150 10679 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:13.273738 9499 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:13.278081 8430 catalog_manager.cc:5484] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "c8d8f771315b46ba84b289cc0189229c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:13.278287 8705 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:13.283463 8835 consensus_queue.cc:237] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:13.285835 9499 raft_consensus.cc:1261] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:15:13.286582 10771 consensus_queue.cc:1035] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:15:13.287389 8772 consensus_peers.cc:485] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c -> Peer 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68:32901): Couldn't send request to peer 9920fbe0481e4ee6960a390ac14e9e5c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24c71780b3154d54b50cf61ec123d6d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:13.289299 8705 raft_consensus.cc:1261] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:15:13.289860 10771 consensus_queue.cc:1035] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:15:13.290454 10771 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:13.292459 9499 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:13.294322 8418 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24c71780b3154d54b50cf61ec123d6d5 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:13.296792 8426 catalog_manager.cc:5484] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c reported cstate change: config changed from index 7 to 8, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New cstate: current_term: 1 leader_uuid: "c8d8f771315b46ba84b289cc0189229c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:13.296574 8705 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:13.476684 10754 ts_tablet_manager.cc:902] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09: Initiating tablet copy from peer f156149e6e7947818ef96621896db45b (127.22.242.66:41441)
I20240503 14:15:13.478040 10754 tablet_copy_client.cc:308] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.66:41441
I20240503 14:15:13.478721 8726 tablet_copy_service.cc:140] P f156149e6e7947818ef96621896db45b: Received BeginTabletCopySession request for tablet a061e8fe907c4f5899d1d90cf1c06754 from peer 85dca8600f3d46b5a5c972c4b343ea09 ({username='slave'} at 127.22.242.69:54561)
I20240503 14:15:13.478884 8726 tablet_copy_service.cc:161] P f156149e6e7947818ef96621896db45b: Beginning new tablet copy session on tablet a061e8fe907c4f5899d1d90cf1c06754 from peer 85dca8600f3d46b5a5c972c4b343ea09 at {username='slave'} at 127.22.242.69:54561: session id = 85dca8600f3d46b5a5c972c4b343ea09-a061e8fe907c4f5899d1d90cf1c06754
I20240503 14:15:13.481932 8726 tablet_copy_source_session.cc:205] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:13.483162 10754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a061e8fe907c4f5899d1d90cf1c06754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:13.487322 10754 tablet_copy_client.cc:783] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:13.487530 10754 tablet_copy_client.cc:647] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:13.489230 10754 tablet_copy_client.cc:523] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:13.491669 10754 tablet_bootstrap.cc:492] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap starting.
I20240503 14:15:13.521795 10754 tablet_bootstrap.cc:492] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:13.522341 10754 tablet_bootstrap.cc:492] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap complete.
I20240503 14:15:13.522588 10754 ts_tablet_manager.cc:1367] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09: Time spent bootstrapping tablet: real 0.031s user 0.028s sys 0.004s
I20240503 14:15:13.523505 10754 raft_consensus.cc:348] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:13.523766 10754 raft_consensus.cc:729] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85dca8600f3d46b5a5c972c4b343ea09, State: Initialized, Role: LEARNER
I20240503 14:15:13.524041 10754 consensus_queue.cc:260] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:13.524657 10754 ts_tablet_manager.cc:1398] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:13.525462 8726 tablet_copy_service.cc:342] P f156149e6e7947818ef96621896db45b: Request end of tablet copy session 85dca8600f3d46b5a5c972c4b343ea09-a061e8fe907c4f5899d1d90cf1c06754 received from {username='slave'} at 127.22.242.69:54561
I20240503 14:15:13.525620 8726 tablet_copy_service.cc:433] P f156149e6e7947818ef96621896db45b: ending tablet copy session 85dca8600f3d46b5a5c972c4b343ea09-a061e8fe907c4f5899d1d90cf1c06754 on tablet a061e8fe907c4f5899d1d90cf1c06754 with peer 85dca8600f3d46b5a5c972c4b343ea09
I20240503 14:15:13.682396 10807 ts_tablet_manager.cc:902] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c: Initiating tablet copy from peer c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:15:13.683524 10807 tablet_copy_client.cc:308] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.67:37569
I20240503 14:15:13.684222 8857 tablet_copy_service.cc:140] P c8d8f771315b46ba84b289cc0189229c: Received BeginTabletCopySession request for tablet 24c71780b3154d54b50cf61ec123d6d5 from peer 9920fbe0481e4ee6960a390ac14e9e5c ({username='slave'} at 127.22.242.68:35055)
I20240503 14:15:13.684376 8857 tablet_copy_service.cc:161] P c8d8f771315b46ba84b289cc0189229c: Beginning new tablet copy session on tablet 24c71780b3154d54b50cf61ec123d6d5 from peer 9920fbe0481e4ee6960a390ac14e9e5c at {username='slave'} at 127.22.242.68:35055: session id = 9920fbe0481e4ee6960a390ac14e9e5c-24c71780b3154d54b50cf61ec123d6d5
I20240503 14:15:13.686180 8857 tablet_copy_source_session.cc:205] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:13.687275 10807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c71780b3154d54b50cf61ec123d6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:13.691823 10807 tablet_copy_client.cc:783] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:13.692081 10807 tablet_copy_client.cc:647] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:13.694216 10807 tablet_copy_client.cc:523] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:13.697571 10807 tablet_bootstrap.cc:492] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap starting.
I20240503 14:15:13.729792 10807 tablet_bootstrap.cc:492] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:13.730209 10807 tablet_bootstrap.cc:492] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap complete.
I20240503 14:15:13.730461 10807 ts_tablet_manager.cc:1367] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.010s
I20240503 14:15:13.731057 10807 raft_consensus.cc:348] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:13.731261 10807 raft_consensus.cc:729] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9920fbe0481e4ee6960a390ac14e9e5c, State: Initialized, Role: LEARNER
I20240503 14:15:13.731462 10807 consensus_queue.cc:260] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:13.731922 10807 ts_tablet_manager.cc:1398] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:15:13.732774 8857 tablet_copy_service.cc:342] P c8d8f771315b46ba84b289cc0189229c: Request end of tablet copy session 9920fbe0481e4ee6960a390ac14e9e5c-24c71780b3154d54b50cf61ec123d6d5 received from {username='slave'} at 127.22.242.68:35055
I20240503 14:15:13.732883 8857 tablet_copy_service.cc:433] P c8d8f771315b46ba84b289cc0189229c: ending tablet copy session 9920fbe0481e4ee6960a390ac14e9e5c-24c71780b3154d54b50cf61ec123d6d5 on tablet 24c71780b3154d54b50cf61ec123d6d5 with peer 9920fbe0481e4ee6960a390ac14e9e5c
I20240503 14:15:13.754501 9499 raft_consensus.cc:1203] T b311c3e036d34d3d8947978df14bae01 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:15:13.919076 9499 raft_consensus.cc:1203] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:15:14.197170 9630 raft_consensus.cc:1203] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:15:14.347486 10863 raft_consensus.cc:1050] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186: attempting to promote NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 to VOTER
I20240503 14:15:14.348275 10863 consensus_queue.cc:237] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:14.350512 9630 raft_consensus.cc:1261] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:15:14.350524 9499 raft_consensus.cc:1261] T b311c3e036d34d3d8947978df14bae01 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:15:14.351136 8705 raft_consensus.cc:1261] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:15:14.351145 10819 consensus_queue.cc:1035] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:15:14.351642 10865 consensus_queue.cc:1035] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:15:14.352044 10863 consensus_queue.cc:1035] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:15:14.354601 10863 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:14.355268 9630 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:14.355404 9499 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:14.359078 8705 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:14.360680 8426 catalog_manager.cc:5484] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: config changed from index 8 to 9, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:14.366956 8572 consensus_queue.cc:237] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:14.368670 9499 raft_consensus.cc:1261] T b311c3e036d34d3d8947978df14bae01 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:15:14.369028 9630 raft_consensus.cc:1261] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:15:14.369210 10819 consensus_queue.cc:1035] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:15:14.369518 10865 consensus_queue.cc:1035] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:15:14.371729 10863 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:14.372344 9630 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:14.372361 9499 raft_consensus.cc:2932] T b311c3e036d34d3d8947978df14bae01 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:14.374608 8419 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b311c3e036d34d3d8947978df14bae01 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:14.375795 8427 catalog_manager.cc:5484] T b311c3e036d34d3d8947978df14bae01 P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: config changed from index 9 to 10, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:14.379374 8685 tablet_service.cc:1514] Processing DeleteTablet for tablet b311c3e036d34d3d8947978df14bae01 with delete_type TABLET_DATA_TOMBSTONED (TS f156149e6e7947818ef96621896db45b not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55852
I20240503 14:15:14.380646 10873 tablet_replica.cc:330] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b: stopping tablet replica
I20240503 14:15:14.380854 10873 raft_consensus.cc:2229] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:14.381307 10873 raft_consensus.cc:2258] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:14.382033 10873 ts_tablet_manager.cc:1875] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:14.386157 10873 ts_tablet_manager.cc:1888] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:15:14.386257 10873 log.cc:1196] T b311c3e036d34d3d8947978df14bae01 P f156149e6e7947818ef96621896db45b: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/b311c3e036d34d3d8947978df14bae01
I20240503 14:15:14.386946 8418 catalog_manager.cc:4830] TS f156149e6e7947818ef96621896db45b (127.22.242.66:41441): tablet b311c3e036d34d3d8947978df14bae01 (table rebalance_test_table_1 [id=9642b97573e44c3783fd3dddeffb9dfe]) successfully deleted
I20240503 14:15:14.519068 10889 raft_consensus.cc:1050] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b: attempting to promote NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 to VOTER
I20240503 14:15:14.519711 10889 consensus_queue.cc:237] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:14.521899 9499 raft_consensus.cc:1261] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:14.522117 8835 raft_consensus.cc:1261] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:14.522622 8572 raft_consensus.cc:1261] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Refusing update from remote peer f156149e6e7947818ef96621896db45b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:14.522550 10891 consensus_queue.cc:1035] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:14.522915 10889 consensus_queue.cc:1035] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:14.523185 10681 consensus_queue.cc:1035] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:14.529495 10891 raft_consensus.cc:2932] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:14.530838 9499 raft_consensus.cc:2932] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:14.532996 8835 raft_consensus.cc:2932] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:14.533139 8572 raft_consensus.cc:2932] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:14.537300 8430 catalog_manager.cc:5484] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b reported cstate change: config changed from index 5 to 6, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f156149e6e7947818ef96621896db45b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:14.624107 10846 raft_consensus.cc:1050] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c: attempting to promote NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c to VOTER
I20240503 14:15:14.624773 10846 consensus_queue.cc:237] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:14.627247 8705 raft_consensus.cc:1261] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:15:14.627288 9630 raft_consensus.cc:1261] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:15:14.627291 9499 raft_consensus.cc:1261] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:15:14.627873 10896 consensus_queue.cc:1035] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:15:14.628271 10846 consensus_queue.cc:1035] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:15:14.628567 10847 consensus_queue.cc:1035] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:15:14.631810 10846 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:14.632490 8705 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:14.632673 9630 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:14.636268 9499 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:14.637897 8427 catalog_manager.cc:5484] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c reported cstate change: config changed from index 8 to 9, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c8d8f771315b46ba84b289cc0189229c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:14.642385 8835 consensus_queue.cc:237] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:14.644047 9499 raft_consensus.cc:1261] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:15:14.644531 9630 raft_consensus.cc:1261] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Refusing update from remote peer c8d8f771315b46ba84b289cc0189229c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:15:14.644538 10846 consensus_queue.cc:1035] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:15:14.645001 10847 consensus_queue.cc:1035] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:15:14.646557 10846 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P c8d8f771315b46ba84b289cc0189229c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:14.647291 9499 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:14.647778 9630 raft_consensus.cc:2932] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:14.650790 8418 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 24c71780b3154d54b50cf61ec123d6d5 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:14.652232 8426 catalog_manager.cc:5484] T 24c71780b3154d54b50cf61ec123d6d5 P 9920fbe0481e4ee6960a390ac14e9e5c reported cstate change: config changed from index 9 to 10, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "c8d8f771315b46ba84b289cc0189229c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:14.655520 8685 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c71780b3154d54b50cf61ec123d6d5 with delete_type TABLET_DATA_TOMBSTONED (TS f156149e6e7947818ef96621896db45b not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55852
I20240503 14:15:14.655885 10873 tablet_replica.cc:330] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b: stopping tablet replica
I20240503 14:15:14.656075 10873 raft_consensus.cc:2229] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:14.656365 10873 raft_consensus.cc:2258] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:14.657204 10873 ts_tablet_manager.cc:1875] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:14.661168 10873 ts_tablet_manager.cc:1888] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:15:14.661284 10873 log.cc:1196] T 24c71780b3154d54b50cf61ec123d6d5 P f156149e6e7947818ef96621896db45b: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/24c71780b3154d54b50cf61ec123d6d5
I20240503 14:15:14.661892 8418 catalog_manager.cc:4830] TS f156149e6e7947818ef96621896db45b (127.22.242.66:41441): tablet 24c71780b3154d54b50cf61ec123d6d5 (table rebalance_test_table_0 [id=1ee49efb58224c89a5d2831f003507e4]) successfully deleted
I20240503 14:15:14.684659 8572 consensus_queue.cc:237] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:14.686625 8835 raft_consensus.cc:1261] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:14.686625 8705 raft_consensus.cc:1261] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:14.687130 10865 consensus_queue.cc:1035] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:14.687402 10863 consensus_queue.cc:1035] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:14.689971 10819 raft_consensus.cc:2932] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } }
I20240503 14:15:14.690536 8705 raft_consensus.cc:2932] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } }
I20240503 14:15:14.691093 8835 raft_consensus.cc:2932] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } }
I20240503 14:15:14.695048 8430 catalog_manager.cc:5484] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } }
I20240503 14:15:14.700889 8572 consensus_queue.cc:237] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:14.703367 8705 raft_consensus.cc:1261] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:14.703676 8835 raft_consensus.cc:1261] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:15:14.703897 8509 consensus_peers.cc:485] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 -> Peer 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68:32901): Couldn't send request to peer 9920fbe0481e4ee6960a390ac14e9e5c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fcba2de60f94132ab750ddc15b8afc5. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:14.704336 10863 consensus_queue.cc:1035] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:14.704643 10819 consensus_queue.cc:1035] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:14.707151 10863 raft_consensus.cc:2932] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:14.708572 8835 raft_consensus.cc:2932] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:14.708693 8705 raft_consensus.cc:2932] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:14.711278 8419 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5fcba2de60f94132ab750ddc15b8afc5 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:14.712941 8433 catalog_manager.cc:5484] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b reported cstate change: config changed from index 4 to 5, NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) added. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } } }
I20240503 14:15:14.738232 8705 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a061e8fe907c4f5899d1d90cf1c06754"
dest_uuid: "f156149e6e7947818ef96621896db45b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52246
I20240503 14:15:14.738413 8705 raft_consensus.cc:595] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Received request to transfer leadership
I20240503 14:15:14.926472 10681 raft_consensus.cc:979] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b: : Instructing follower 85dca8600f3d46b5a5c972c4b343ea09 to start an election
I20240503 14:15:14.926666 10681 raft_consensus.cc:1067] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Signalling peer 85dca8600f3d46b5a5c972c4b343ea09 to start an election
I20240503 14:15:14.927363 9499 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a061e8fe907c4f5899d1d90cf1c06754"
dest_uuid: "85dca8600f3d46b5a5c972c4b343ea09"
from {username='slave'} at 127.22.242.66:51551
I20240503 14:15:14.927639 9499 raft_consensus.cc:482] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:15:14.927757 9499 raft_consensus.cc:3037] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:14.929862 9499 raft_consensus.cc:504] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:14.931411 9499 leader_election.cc:288] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [CANDIDATE]: Term 2 election: Requested vote from peers f156149e6e7947818ef96621896db45b (127.22.242.66:41441), 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379), c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569)
I20240503 14:15:14.931727 8705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a061e8fe907c4f5899d1d90cf1c06754" candidate_uuid: "85dca8600f3d46b5a5c972c4b343ea09" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "f156149e6e7947818ef96621896db45b"
I20240503 14:15:14.931900 8705 raft_consensus.cc:3032] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:15:14.931859 8572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a061e8fe907c4f5899d1d90cf1c06754" candidate_uuid: "85dca8600f3d46b5a5c972c4b343ea09" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "6aab696fc59c4784a0814eadd67ae186"
I20240503 14:15:14.931972 8705 raft_consensus.cc:729] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f156149e6e7947818ef96621896db45b, State: Running, Role: LEADER
I20240503 14:15:14.932086 8572 raft_consensus.cc:3037] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:14.932101 8835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a061e8fe907c4f5899d1d90cf1c06754" candidate_uuid: "85dca8600f3d46b5a5c972c4b343ea09" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "c8d8f771315b46ba84b289cc0189229c"
I20240503 14:15:14.932224 8705 consensus_queue.cc:260] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:14.932358 8835 raft_consensus.cc:3037] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:14.932524 8705 raft_consensus.cc:3037] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:14.934559 8705 raft_consensus.cc:2454] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85dca8600f3d46b5a5c972c4b343ea09 in term 2.
I20240503 14:15:14.935060 8572 raft_consensus.cc:2454] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85dca8600f3d46b5a5c972c4b343ea09 in term 2.
I20240503 14:15:14.935058 8835 raft_consensus.cc:2454] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85dca8600f3d46b5a5c972c4b343ea09 in term 2.
I20240503 14:15:14.935562 9433 leader_election.cc:302] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 85dca8600f3d46b5a5c972c4b343ea09, c8d8f771315b46ba84b289cc0189229c, f156149e6e7947818ef96621896db45b; no voters:
I20240503 14:15:14.935921 10658 raft_consensus.cc:2781] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:14.936347 10658 raft_consensus.cc:686] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [term 2 LEADER]: Becoming Leader. State: Replica: 85dca8600f3d46b5a5c972c4b343ea09, State: Running, Role: LEADER
I20240503 14:15:14.936781 10658 consensus_queue.cc:237] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:14.940125 8430 catalog_manager.cc:5484] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 reported cstate change: term changed from 1 to 2, leader changed from f156149e6e7947818ef96621896db45b (127.22.242.66) to 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69). New cstate: current_term: 2 leader_uuid: "85dca8600f3d46b5a5c972c4b343ea09" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:14.949708 8572 consensus_queue.cc:237] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:14.951794 8705 raft_consensus.cc:1261] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:14.951922 9630 raft_consensus.cc:1261] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:14.952399 10865 consensus_queue.cc:1035] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:14.952692 10863 consensus_queue.cc:1035] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:14.955329 10865 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:14.955983 9630 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:14.956072 8705 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:14.959760 8430 catalog_manager.cc:5484] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:14.965342 8572 consensus_queue.cc:237] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: true } }
I20240503 14:15:14.967547 9630 raft_consensus.cc:1261] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 14:15:14.968250 8508 consensus_peers.cc:485] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 -> Peer c8d8f771315b46ba84b289cc0189229c (127.22.242.67:37569): Couldn't send request to peer c8d8f771315b46ba84b289cc0189229c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:14.968354 8705 raft_consensus.cc:1261] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:15:14.968933 10865 consensus_queue.cc:1035] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:15:14.969409 10819 consensus_queue.cc:1035] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:15:14.971390 10865 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8d8f771315b46ba84b289cc0189229c (127.22.242.67) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: true } } }
I20240503 14:15:14.972256 8705 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8d8f771315b46ba84b289cc0189229c (127.22.242.67) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: true } } }
I20240503 14:15:14.972011 9630 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8d8f771315b46ba84b289cc0189229c (127.22.242.67) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: true } } }
I20240503 14:15:14.975643 8419 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 029f0fef59a740f88a26ee36fa15907f with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:14.976246 8433 catalog_manager.cc:5484] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b reported cstate change: config changed from index 7 to 8, NON_VOTER c8d8f771315b46ba84b289cc0189229c (127.22.242.67) added. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: true } } }
I20240503 14:15:15.159121 10953 ts_tablet_manager.cc:902] T 5fcba2de60f94132ab750ddc15b8afc5 P 9920fbe0481e4ee6960a390ac14e9e5c: Initiating tablet copy from peer 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379)
I20240503 14:15:15.160563 10953 tablet_copy_client.cc:308] T 5fcba2de60f94132ab750ddc15b8afc5 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:36379
I20240503 14:15:15.161237 8594 tablet_copy_service.cc:140] P 6aab696fc59c4784a0814eadd67ae186: Received BeginTabletCopySession request for tablet 5fcba2de60f94132ab750ddc15b8afc5 from peer 9920fbe0481e4ee6960a390ac14e9e5c ({username='slave'} at 127.22.242.68:60801)
I20240503 14:15:15.161417 8594 tablet_copy_service.cc:161] P 6aab696fc59c4784a0814eadd67ae186: Beginning new tablet copy session on tablet 5fcba2de60f94132ab750ddc15b8afc5 from peer 9920fbe0481e4ee6960a390ac14e9e5c at {username='slave'} at 127.22.242.68:60801: session id = 9920fbe0481e4ee6960a390ac14e9e5c-5fcba2de60f94132ab750ddc15b8afc5
I20240503 14:15:15.163455 8594 tablet_copy_source_session.cc:205] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:15.164749 10953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcba2de60f94132ab750ddc15b8afc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:15.168927 10953 tablet_copy_client.cc:783] T 5fcba2de60f94132ab750ddc15b8afc5 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:15.169181 10953 tablet_copy_client.cc:647] T 5fcba2de60f94132ab750ddc15b8afc5 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:15.170653 10953 tablet_copy_client.cc:523] T 5fcba2de60f94132ab750ddc15b8afc5 P 9920fbe0481e4ee6960a390ac14e9e5c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:15.172842 10953 tablet_bootstrap.cc:492] T 5fcba2de60f94132ab750ddc15b8afc5 P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap starting.
I20240503 14:15:15.204388 10953 tablet_bootstrap.cc:492] T 5fcba2de60f94132ab750ddc15b8afc5 P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:15.204882 10953 tablet_bootstrap.cc:492] T 5fcba2de60f94132ab750ddc15b8afc5 P 9920fbe0481e4ee6960a390ac14e9e5c: Bootstrap complete.
I20240503 14:15:15.205178 10953 ts_tablet_manager.cc:1367] T 5fcba2de60f94132ab750ddc15b8afc5 P 9920fbe0481e4ee6960a390ac14e9e5c: Time spent bootstrapping tablet: real 0.032s user 0.031s sys 0.001s
I20240503 14:15:15.205890 10953 raft_consensus.cc:348] T 5fcba2de60f94132ab750ddc15b8afc5 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:15.206086 10953 raft_consensus.cc:729] T 5fcba2de60f94132ab750ddc15b8afc5 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9920fbe0481e4ee6960a390ac14e9e5c, State: Initialized, Role: LEARNER
I20240503 14:15:15.206285 10953 consensus_queue.cc:260] T 5fcba2de60f94132ab750ddc15b8afc5 P 9920fbe0481e4ee6960a390ac14e9e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: NON_VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: true } }
I20240503 14:15:15.206825 10953 ts_tablet_manager.cc:1398] T 5fcba2de60f94132ab750ddc15b8afc5 P 9920fbe0481e4ee6960a390ac14e9e5c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:15:15.208387 8594 tablet_copy_service.cc:342] P 6aab696fc59c4784a0814eadd67ae186: Request end of tablet copy session 9920fbe0481e4ee6960a390ac14e9e5c-5fcba2de60f94132ab750ddc15b8afc5 received from {username='slave'} at 127.22.242.68:60801
I20240503 14:15:15.208508 8594 tablet_copy_service.cc:433] P 6aab696fc59c4784a0814eadd67ae186: ending tablet copy session 9920fbe0481e4ee6960a390ac14e9e5c-5fcba2de60f94132ab750ddc15b8afc5 on tablet 5fcba2de60f94132ab750ddc15b8afc5 with peer 9920fbe0481e4ee6960a390ac14e9e5c
I20240503 14:15:15.333314 8572 raft_consensus.cc:1261] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 [term 2 FOLLOWER]: Refusing update from remote peer 85dca8600f3d46b5a5c972c4b343ea09: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:15:15.333966 10658 consensus_queue.cc:1035] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:15.472981 8835 raft_consensus.cc:1261] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c [term 2 FOLLOWER]: Refusing update from remote peer 85dca8600f3d46b5a5c972c4b343ea09: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:15:15.473611 10658 consensus_queue.cc:1035] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:15.478880 8705 raft_consensus.cc:1261] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 2 FOLLOWER]: Refusing update from remote peer 85dca8600f3d46b5a5c972c4b343ea09: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:15:15.480520 10658 consensus_queue.cc:1035] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:15.503475 9499 consensus_queue.cc:237] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:15.505308 8835 raft_consensus.cc:1261] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c [term 2 FOLLOWER]: Refusing update from remote peer 85dca8600f3d46b5a5c972c4b343ea09: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:15:15.505283 8572 raft_consensus.cc:1261] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 [term 2 FOLLOWER]: Refusing update from remote peer 85dca8600f3d46b5a5c972c4b343ea09: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:15:15.505842 10991 consensus_queue.cc:1035] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:15:15.506124 10658 consensus_queue.cc:1035] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:15:15.508301 10658 raft_consensus.cc:2932] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:15.509065 8572 raft_consensus.cc:2932] T a061e8fe907c4f5899d1d90cf1c06754 P 6aab696fc59c4784a0814eadd67ae186 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:15.509274 8835 raft_consensus.cc:2932] T a061e8fe907c4f5899d1d90cf1c06754 P c8d8f771315b46ba84b289cc0189229c [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:15.511492 8417 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a061e8fe907c4f5899d1d90cf1c06754 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:15.513231 8426 catalog_manager.cc:5484] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 reported cstate change: config changed from index 6 to 8, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New cstate: current_term: 2 leader_uuid: "85dca8600f3d46b5a5c972c4b343ea09" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:15.517087 10999 ts_tablet_manager.cc:902] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: Initiating tablet copy from peer 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379)
I20240503 14:15:15.519199 10999 tablet_copy_client.cc:308] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:36379
I20240503 14:15:15.520149 8594 tablet_copy_service.cc:140] P 6aab696fc59c4784a0814eadd67ae186: Received BeginTabletCopySession request for tablet 029f0fef59a740f88a26ee36fa15907f from peer c8d8f771315b46ba84b289cc0189229c ({username='slave'} at 127.22.242.67:55401)
I20240503 14:15:15.520359 8594 tablet_copy_service.cc:161] P 6aab696fc59c4784a0814eadd67ae186: Beginning new tablet copy session on tablet 029f0fef59a740f88a26ee36fa15907f from peer c8d8f771315b46ba84b289cc0189229c at {username='slave'} at 127.22.242.67:55401: session id = c8d8f771315b46ba84b289cc0189229c-029f0fef59a740f88a26ee36fa15907f
I20240503 14:15:15.522940 8594 tablet_copy_source_session.cc:205] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:15.524293 10999 ts_tablet_manager.cc:1875] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:15:15.524578 8685 tablet_service.cc:1514] Processing DeleteTablet for tablet a061e8fe907c4f5899d1d90cf1c06754 with delete_type TABLET_DATA_TOMBSTONED (TS f156149e6e7947818ef96621896db45b not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:55852
I20240503 14:15:15.525249 11005 tablet_replica.cc:330] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b: stopping tablet replica
I20240503 14:15:15.525444 11005 raft_consensus.cc:2229] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:15.525681 11005 raft_consensus.cc:2258] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:15.526571 11005 ts_tablet_manager.cc:1875] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:15.529145 10999 ts_tablet_manager.cc:1888] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 14:15:15.529448 10999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 029f0fef59a740f88a26ee36fa15907f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:15.530929 11005 ts_tablet_manager.cc:1888] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 14:15:15.531033 11005 log.cc:1196] T a061e8fe907c4f5899d1d90cf1c06754 P f156149e6e7947818ef96621896db45b: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/a061e8fe907c4f5899d1d90cf1c06754
I20240503 14:15:15.531737 8418 catalog_manager.cc:4830] TS f156149e6e7947818ef96621896db45b (127.22.242.66:41441): tablet a061e8fe907c4f5899d1d90cf1c06754 (table rebalance_test_table_0 [id=1ee49efb58224c89a5d2831f003507e4]) successfully deleted
I20240503 14:15:15.531788 10999 tablet_copy_client.cc:783] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:15.532028 10999 tablet_copy_client.cc:647] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:15.533979 10999 tablet_copy_client.cc:523] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:15.536639 10999 tablet_bootstrap.cc:492] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: Bootstrap starting.
I20240503 14:15:15.570387 10999 tablet_bootstrap.cc:492] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:15.570794 10999 tablet_bootstrap.cc:492] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: Bootstrap complete.
I20240503 14:15:15.571028 10999 ts_tablet_manager.cc:1367] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: Time spent bootstrapping tablet: real 0.034s user 0.032s sys 0.000s
I20240503 14:15:15.571691 10999 raft_consensus.cc:348] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: true } }
I20240503 14:15:15.571909 10999 raft_consensus.cc:729] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8d8f771315b46ba84b289cc0189229c, State: Initialized, Role: LEARNER
I20240503 14:15:15.572062 10999 consensus_queue.cc:260] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: NON_VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: true } }
I20240503 14:15:15.572604 10999 ts_tablet_manager.cc:1398] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 14:15:15.573483 8594 tablet_copy_service.cc:342] P 6aab696fc59c4784a0814eadd67ae186: Request end of tablet copy session c8d8f771315b46ba84b289cc0189229c-029f0fef59a740f88a26ee36fa15907f received from {username='slave'} at 127.22.242.67:55401
I20240503 14:15:15.573613 8594 tablet_copy_service.cc:433] P 6aab696fc59c4784a0814eadd67ae186: ending tablet copy session c8d8f771315b46ba84b289cc0189229c-029f0fef59a740f88a26ee36fa15907f on tablet 029f0fef59a740f88a26ee36fa15907f with peer c8d8f771315b46ba84b289cc0189229c
I20240503 14:15:15.644308 9630 raft_consensus.cc:1203] T 5fcba2de60f94132ab750ddc15b8afc5 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:15:15.899987 8572 consensus_queue.cc:237] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } }
I20240503 14:15:15.902040 8835 raft_consensus.cc:1261] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:15.902132 8705 raft_consensus.cc:1261] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:15.902688 11007 consensus_queue.cc:1035] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:15.902971 11008 consensus_queue.cc:1035] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:15.905622 10819 raft_consensus.cc:2932] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } }
I20240503 14:15:15.906417 8835 raft_consensus.cc:2932] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } }
I20240503 14:15:15.906343 8705 raft_consensus.cc:2932] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } }
I20240503 14:15:15.911401 8433 catalog_manager.cc:5484] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } }
I20240503 14:15:15.916544 8572 consensus_queue.cc:237] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
W20240503 14:15:15.918953 8507 consensus_peers.cc:485] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 -> Peer 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69:41037): Couldn't send request to peer 85dca8600f3d46b5a5c972c4b343ea09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 610307a610a4495382bf62b26477a5db. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:15.919492 8835 raft_consensus.cc:1261] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:15.919652 8705 raft_consensus.cc:1261] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:15.920056 10819 consensus_queue.cc:1035] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:15.920334 11008 consensus_queue.cc:1035] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:15.922683 11008 raft_consensus.cc:2932] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:15.923301 8706 raft_consensus.cc:2932] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:15.925972 8419 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 610307a610a4495382bf62b26477a5db with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:15.928292 8433 catalog_manager.cc:5484] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: config changed from index 4 to 5, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:15.929863 8835 raft_consensus.cc:2932] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } } }
I20240503 14:15:16.012956 8835 raft_consensus.cc:1203] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:15:16.249408 11008 raft_consensus.cc:1050] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186: attempting to promote NON_VOTER 9920fbe0481e4ee6960a390ac14e9e5c to VOTER
I20240503 14:15:16.250056 11008 consensus_queue.cc:237] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:16.252328 8835 raft_consensus.cc:1261] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:16.252318 9630 raft_consensus.cc:1261] T 5fcba2de60f94132ab750ddc15b8afc5 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 LEARNER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:16.252492 8706 raft_consensus.cc:1261] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:16.252910 11008 consensus_queue.cc:1035] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:16.253261 11064 consensus_queue.cc:1035] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:16.253541 10819 consensus_queue.cc:1035] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:16.256188 10819 raft_consensus.cc:2932] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:16.256748 8835 raft_consensus.cc:2932] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:16.260638 9630 raft_consensus.cc:2932] T 5fcba2de60f94132ab750ddc15b8afc5 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:16.261826 8430 catalog_manager.cc:5484] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: config changed from index 5 to 6, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:16.263978 8706 raft_consensus.cc:2932] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9920fbe0481e4ee6960a390ac14e9e5c (127.22.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:16.266901 8572 consensus_queue.cc:237] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } }
I20240503 14:15:16.268792 9630 raft_consensus.cc:1261] T 5fcba2de60f94132ab750ddc15b8afc5 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:16.269057 8835 raft_consensus.cc:1261] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:16.269531 10819 consensus_queue.cc:1035] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:16.269778 11007 consensus_queue.cc:1035] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:16.271849 11007 raft_consensus.cc:2932] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:16.272411 8835 raft_consensus.cc:2932] T 5fcba2de60f94132ab750ddc15b8afc5 P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:16.273314 9630 raft_consensus.cc:2932] T 5fcba2de60f94132ab750ddc15b8afc5 P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } }
I20240503 14:15:16.275308 8419 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5fcba2de60f94132ab750ddc15b8afc5 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:16.276144 8430 catalog_manager.cc:5484] T 5fcba2de60f94132ab750ddc15b8afc5 P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: config changed from index 6 to 7, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:16.280727 8685 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fcba2de60f94132ab750ddc15b8afc5 with delete_type TABLET_DATA_TOMBSTONED (TS f156149e6e7947818ef96621896db45b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55852
I20240503 14:15:16.281391 11069 tablet_replica.cc:330] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b: stopping tablet replica
I20240503 14:15:16.281591 11069 raft_consensus.cc:2229] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:16.281924 11069 raft_consensus.cc:2258] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:16.282776 11069 ts_tablet_manager.cc:1875] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:16.286810 11069 ts_tablet_manager.cc:1888] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:15:16.286926 11069 log.cc:1196] T 5fcba2de60f94132ab750ddc15b8afc5 P f156149e6e7947818ef96621896db45b: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/5fcba2de60f94132ab750ddc15b8afc5
I20240503 14:15:16.287602 8418 catalog_manager.cc:4830] TS f156149e6e7947818ef96621896db45b (127.22.242.66:41441): tablet 5fcba2de60f94132ab750ddc15b8afc5 (table rebalance_test_table_1 [id=9642b97573e44c3783fd3dddeffb9dfe]) successfully deleted
I20240503 14:15:16.485263 11085 ts_tablet_manager.cc:902] T 610307a610a4495382bf62b26477a5db P 85dca8600f3d46b5a5c972c4b343ea09: Initiating tablet copy from peer 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379)
I20240503 14:15:16.486380 11085 tablet_copy_client.cc:308] T 610307a610a4495382bf62b26477a5db P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Beginning tablet copy session from remote peer at address 127.22.242.65:36379
I20240503 14:15:16.486991 8594 tablet_copy_service.cc:140] P 6aab696fc59c4784a0814eadd67ae186: Received BeginTabletCopySession request for tablet 610307a610a4495382bf62b26477a5db from peer 85dca8600f3d46b5a5c972c4b343ea09 ({username='slave'} at 127.22.242.69:52357)
I20240503 14:15:16.487156 8594 tablet_copy_service.cc:161] P 6aab696fc59c4784a0814eadd67ae186: Beginning new tablet copy session on tablet 610307a610a4495382bf62b26477a5db from peer 85dca8600f3d46b5a5c972c4b343ea09 at {username='slave'} at 127.22.242.69:52357: session id = 85dca8600f3d46b5a5c972c4b343ea09-610307a610a4495382bf62b26477a5db
I20240503 14:15:16.489182 8594 tablet_copy_source_session.cc:205] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:16.490240 11085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610307a610a4495382bf62b26477a5db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:16.494433 11085 tablet_copy_client.cc:783] T 610307a610a4495382bf62b26477a5db P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:16.494683 11085 tablet_copy_client.cc:647] T 610307a610a4495382bf62b26477a5db P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:16.496294 11085 tablet_copy_client.cc:523] T 610307a610a4495382bf62b26477a5db P 85dca8600f3d46b5a5c972c4b343ea09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:16.498509 11085 tablet_bootstrap.cc:492] T 610307a610a4495382bf62b26477a5db P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap starting.
I20240503 14:15:16.529963 11085 tablet_bootstrap.cc:492] T 610307a610a4495382bf62b26477a5db P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:16.530368 11085 tablet_bootstrap.cc:492] T 610307a610a4495382bf62b26477a5db P 85dca8600f3d46b5a5c972c4b343ea09: Bootstrap complete.
I20240503 14:15:16.530622 11085 ts_tablet_manager.cc:1367] T 610307a610a4495382bf62b26477a5db P 85dca8600f3d46b5a5c972c4b343ea09: Time spent bootstrapping tablet: real 0.032s user 0.032s sys 0.000s
I20240503 14:15:16.531282 11085 raft_consensus.cc:348] T 610307a610a4495382bf62b26477a5db P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:16.531504 11085 raft_consensus.cc:729] T 610307a610a4495382bf62b26477a5db P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85dca8600f3d46b5a5c972c4b343ea09, State: Initialized, Role: LEARNER
I20240503 14:15:16.531692 11085 consensus_queue.cc:260] T 610307a610a4495382bf62b26477a5db P 85dca8600f3d46b5a5c972c4b343ea09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: NON_VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: true } }
I20240503 14:15:16.532135 11085 ts_tablet_manager.cc:1398] T 610307a610a4495382bf62b26477a5db P 85dca8600f3d46b5a5c972c4b343ea09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:15:16.532917 8594 tablet_copy_service.cc:342] P 6aab696fc59c4784a0814eadd67ae186: Request end of tablet copy session 85dca8600f3d46b5a5c972c4b343ea09-610307a610a4495382bf62b26477a5db received from {username='slave'} at 127.22.242.69:52357
I20240503 14:15:16.533036 8594 tablet_copy_service.cc:433] P 6aab696fc59c4784a0814eadd67ae186: ending tablet copy session 85dca8600f3d46b5a5c972c4b343ea09-610307a610a4495382bf62b26477a5db on tablet 610307a610a4495382bf62b26477a5db with peer 85dca8600f3d46b5a5c972c4b343ea09
I20240503 14:15:16.615289 11007 raft_consensus.cc:1050] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186: attempting to promote NON_VOTER c8d8f771315b46ba84b289cc0189229c to VOTER
I20240503 14:15:16.616024 11007 consensus_queue.cc:237] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: false } }
I20240503 14:15:16.618427 8835 raft_consensus.cc:1261] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 1 LEARNER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:15:16.618402 9630 raft_consensus.cc:1261] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:15:16.618482 8706 raft_consensus.cc:1261] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:15:16.619024 11068 consensus_queue.cc:1035] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:15:16.619428 10819 consensus_queue.cc:1035] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:15:16.619678 11064 consensus_queue.cc:1035] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:15:16.630219 11008 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c8d8f771315b46ba84b289cc0189229c (127.22.242.67) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: false } } }
I20240503 14:15:16.630934 8835 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c8d8f771315b46ba84b289cc0189229c (127.22.242.67) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: false } } }
I20240503 14:15:16.630954 9630 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c8d8f771315b46ba84b289cc0189229c (127.22.242.67) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: false } } }
I20240503 14:15:16.636837 8426 catalog_manager.cc:5484] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c reported cstate change: config changed from index 8 to 9, c8d8f771315b46ba84b289cc0189229c (127.22.242.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: false } } }
I20240503 14:15:16.641108 8706 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c8d8f771315b46ba84b289cc0189229c (127.22.242.67) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: false } } }
I20240503 14:15:16.647420 8572 consensus_queue.cc:237] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: false } }
I20240503 14:15:16.650051 8835 raft_consensus.cc:1261] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:15:16.650233 9630 raft_consensus.cc:1261] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:15:16.650803 11008 consensus_queue.cc:1035] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:15:16.651731 11064 consensus_queue.cc:1035] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:15:16.654223 11064 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: false } } }
I20240503 14:15:16.656831 8835 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: false } } }
I20240503 14:15:16.657673 8419 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 029f0fef59a740f88a26ee36fa15907f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:16.658541 9630 raft_consensus.cc:2932] T 029f0fef59a740f88a26ee36fa15907f P 9920fbe0481e4ee6960a390ac14e9e5c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: false } } }
I20240503 14:15:16.659307 8426 catalog_manager.cc:5484] T 029f0fef59a740f88a26ee36fa15907f P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: config changed from index 9 to 10, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9920fbe0481e4ee6960a390ac14e9e5c" member_type: VOTER last_known_addr { host: "127.22.242.68" port: 32901 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:16.662763 8685 tablet_service.cc:1514] Processing DeleteTablet for tablet 029f0fef59a740f88a26ee36fa15907f with delete_type TABLET_DATA_TOMBSTONED (TS f156149e6e7947818ef96621896db45b not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55852
I20240503 14:15:16.663201 11069 tablet_replica.cc:330] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b: stopping tablet replica
I20240503 14:15:16.663463 11069 raft_consensus.cc:2229] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:16.663969 11069 raft_consensus.cc:2258] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:16.665544 11069 ts_tablet_manager.cc:1875] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:16.670188 11069 ts_tablet_manager.cc:1888] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:15:16.670295 11069 log.cc:1196] T 029f0fef59a740f88a26ee36fa15907f P f156149e6e7947818ef96621896db45b: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/029f0fef59a740f88a26ee36fa15907f
I20240503 14:15:16.670969 8418 catalog_manager.cc:4830] TS f156149e6e7947818ef96621896db45b (127.22.242.66:41441): tablet 029f0fef59a740f88a26ee36fa15907f (table rebalance_test_table_2 [id=ca152a2c2e694184b98c458650a677ee]) successfully deleted
I20240503 14:15:16.861716 9499 raft_consensus.cc:1203] T 610307a610a4495382bf62b26477a5db P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:15:17.419112 11125 raft_consensus.cc:1050] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186: attempting to promote NON_VOTER 85dca8600f3d46b5a5c972c4b343ea09 to VOTER
I20240503 14:15:17.419720 11125 consensus_queue.cc:237] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:17.422065 8835 raft_consensus.cc:1261] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:17.422327 9499 raft_consensus.cc:1261] T 610307a610a4495382bf62b26477a5db P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 LEARNER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:17.422560 8706 raft_consensus.cc:1261] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:17.422685 11127 consensus_queue.cc:1035] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:17.423091 11125 consensus_queue.cc:1035] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:17.423449 11064 consensus_queue.cc:1035] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:17.426414 11125 raft_consensus.cc:2932] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:17.427217 9499 raft_consensus.cc:2932] T 610307a610a4495382bf62b26477a5db P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:17.430719 8706 raft_consensus.cc:2932] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:17.431227 8426 catalog_manager.cc:5484] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: config changed from index 5 to 6, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:17.440541 8835 raft_consensus.cc:2932] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85dca8600f3d46b5a5c972c4b343ea09 (127.22.242.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f156149e6e7947818ef96621896db45b" member_type: VOTER last_known_addr { host: "127.22.242.66" port: 41441 } attrs { replace: true } } peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:17.446144 8572 consensus_queue.cc:237] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } }
I20240503 14:15:17.448143 8835 raft_consensus.cc:1261] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:17.448134 9499 raft_consensus.cc:1261] T 610307a610a4495382bf62b26477a5db P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Refusing update from remote peer 6aab696fc59c4784a0814eadd67ae186: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:17.448706 11127 consensus_queue.cc:1035] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:17.449421 11127 consensus_queue.cc:1035] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:17.451401 11125 raft_consensus.cc:2932] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:17.452126 8835 raft_consensus.cc:2932] T 610307a610a4495382bf62b26477a5db P c8d8f771315b46ba84b289cc0189229c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:17.455355 8419 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 610307a610a4495382bf62b26477a5db with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:17.456838 8426 catalog_manager.cc:5484] T 610307a610a4495382bf62b26477a5db P 6aab696fc59c4784a0814eadd67ae186 reported cstate change: config changed from index 6 to 7, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New cstate: current_term: 1 leader_uuid: "6aab696fc59c4784a0814eadd67ae186" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:17.457445 9499 raft_consensus.cc:2932] T 610307a610a4495382bf62b26477a5db P 85dca8600f3d46b5a5c972c4b343ea09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f156149e6e7947818ef96621896db45b (127.22.242.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6aab696fc59c4784a0814eadd67ae186" member_type: VOTER last_known_addr { host: "127.22.242.65" port: 36379 } } peers { permanent_uuid: "c8d8f771315b46ba84b289cc0189229c" member_type: VOTER last_known_addr { host: "127.22.242.67" port: 37569 } } peers { permanent_uuid: "85dca8600f3d46b5a5c972c4b343ea09" member_type: VOTER last_known_addr { host: "127.22.242.69" port: 41037 } attrs { promote: false } } }
I20240503 14:15:17.460522 8685 tablet_service.cc:1514] Processing DeleteTablet for tablet 610307a610a4495382bf62b26477a5db with delete_type TABLET_DATA_TOMBSTONED (TS f156149e6e7947818ef96621896db45b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55852
I20240503 14:15:17.471060 11135 tablet_replica.cc:330] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b: stopping tablet replica
I20240503 14:15:17.471217 11135 raft_consensus.cc:2229] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:17.471482 11135 raft_consensus.cc:2258] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:17.472419 11135 ts_tablet_manager.cc:1875] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:17.476809 11135 ts_tablet_manager.cc:1888] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:15:17.476938 11135 log.cc:1196] T 610307a610a4495382bf62b26477a5db P f156149e6e7947818ef96621896db45b: Deleting WAL directory at /tmp/dist-test-taskqfBHnl/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714745515241910-23497-0/raft_consensus-itest-cluster/ts-1/wal/wals/610307a610a4495382bf62b26477a5db
I20240503 14:15:17.477687 8418 catalog_manager.cc:4830] TS f156149e6e7947818ef96621896db45b (127.22.242.66:41441): tablet 610307a610a4495382bf62b26477a5db (table rebalance_test_table_2 [id=ca152a2c2e694184b98c458650a677ee]) successfully deleted
I20240503 14:15:17.825894 8817 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:17.826009 9479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:17.830960 8553 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:17.835327 9611 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:17.840338 8685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:18.002372 8553 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:18.004694 9479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:18.018390 9611 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:18.018481 8685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:18.042944 8817 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:18.184263 8553 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:18.199106 8817 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:18.201920 8685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:18.209251 9611 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:18.211277 9479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:18.372599 8553 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:18.384268 8817 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:18.388509 9479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:18.391603 9611 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:18.400789 8685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:18.564599 8553 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:18.567214 9479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:18.573012 9611 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:18.579824 8817 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:18.591400 8685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:18.791376 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 8492
I20240503 14:15:18.798812 8614 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:18.957249 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 8492
W20240503 14:15:18.984823 9564 connection.cc:535] server connection from 127.22.242.65:58611 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:18.985162 8772 connection.cc:535] server connection from 127.22.242.65:50395 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:15:18.985852 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 8624
I20240503 14:15:18.992085 8745 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:19.153123 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 8624
I20240503 14:15:19.183691 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 8755
I20240503 14:15:19.195909 8876 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:15:19.310412 9434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.242.65:36379: connect: Connection refused (error 111)
W20240503 14:15:19.311820 9434 consensus_peers.cc:485] T a061e8fe907c4f5899d1d90cf1c06754 P 85dca8600f3d46b5a5c972c4b343ea09 -> Peer 6aab696fc59c4784a0814eadd67ae186 (127.22.242.65:36379): Couldn't send request to peer 6aab696fc59c4784a0814eadd67ae186. Status: Network error: Client connection negotiation failed: client connection to 127.22.242.65:36379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:19.345793 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 8755
I20240503 14:15:19.372459 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 9548
I20240503 14:15:19.378417 9670 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:19.513976 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 9548
I20240503 14:15:19.538517 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 9417
I20240503 14:15:19.544772 9538 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:19.680022 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 9417
I20240503 14:15:19.704056 23497 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu pid 8402
I20240503 14:15:19.705202 8463 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:19.862174 23497 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqfBHnl/build/asan/bin/kudu with pid 8402
2024-05-03T14:15:19Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2 (29563 ms)
[----------] 2 tests from LocationAwareRebalancingParamTest (54993 ms total)
[----------] Global test environment tear-down
[==========] 7 tests from 6 test suites ran. (204685 ms total)
[ PASSED ] 6 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1, where GetParam() = 4-byte object <01-00 00-00>
1 FAILED TEST